[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 591897573 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002368] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007550] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013104] *** VALIDATE selinux *** [ 0.026000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031367] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032263] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035180] *** VALIDATE tmpfs *** [ 0.036797] *** VALIDATE proc *** [ 0.037319] *** VALIDATE cgroup *** [ 0.038007] *** VALIDATE cgroup2 *** [ 0.040042] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042251] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044047] Spectre V2 : User space: Vulnerable [ 0.045015] Speculative Store Bypass: Vulnerable [ 0.050375] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.053806] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055029] ... version: 2 [ 0.056011] ... bit width: 48 [ 0.057012] ... generic registers: 4 [ 0.058012] ... value mask: 0000ffffffffffff [ 0.059020] ... max period: 00007fffffffffff [ 0.060013] ... fixed-purpose events: 3 [ 0.061009] ... event mask: 000000070000000f [ 0.062314] rcu: Hierarchical SRCU implementation. [ 0.064717] smp: Bringing up secondary CPUs ... [ 0.065669] x86: Booting SMP configuration: [ 0.066019] .... node #0, CPUs: #1 #2 #3 [ 0.099783] smp: Brought up 1 node, 4 CPUs [ 0.101011] smpboot: Max logical packages: 1 [ 0.102013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128014] node 0 deferred pages initialised in 21ms [ 0.134010] devtmpfs: initialized [ 0.135286] x86/mm: Memory block size: 128MB [ 0.138185] gcov: version magic: 0x41383552 [ 0.141446] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143270] pinctrl core: initialized pinctrl subsystem [ 0.144250] [ 0.144904] ************************************************************* [ 0.145014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146014] ** ** [ 0.147011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148011] ** ** [ 0.149017] ** This means that this kernel is built to expose internal ** [ 0.150011] ** IOMMU data structures, which may compromise security on ** [ 0.151009] ** your system. ** [ 0.152013] ** ** [ 0.153016] ** If you see this message and you are not debugging the ** [ 0.154012] ** kernel, report this immediately to your vendor! ** [ 0.155014] ** ** [ 0.156011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157014] ************************************************************* [ 0.159564] NET: Registered protocol family 16 [ 0.160626] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.163452] cpuidle: using governor menu [ 0.165458] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170462] PCI: Using configuration type 1 for base access [ 0.173911] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186042] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195090] cryptd: max_cpu_qlen set to 1000 [ 0.198625] ACPI: Added _OSI(Module Device) [ 0.199016] ACPI: Added _OSI(Processor Device) [ 0.200018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206445] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210822] ACPI: Interpreter enabled [ 0.213063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214012] ACPI: Using IOAPIC for interrupt routing [ 0.216143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229377] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.236020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246805] acpiphp: Slot [2] registered [ 0.247172] acpiphp: Slot [3] registered [ 0.249067] acpiphp: Slot [4] registered [ 0.251133] acpiphp: Slot [5] registered [ 0.252097] acpiphp: Slot [6] registered [ 0.254227] acpiphp: Slot [7] registered [ 0.256080] acpiphp: Slot [8] registered [ 0.258088] acpiphp: Slot [9] registered [ 0.260021] acpiphp: Slot [10] registered [ 0.261121] acpiphp: Slot [11] registered [ 0.262915] acpiphp: Slot [12] registered [ 0.264114] acpiphp: Slot [13] registered [ 0.267109] acpiphp: Slot [14] registered [ 0.269132] acpiphp: Slot [15] registered [ 0.271112] acpiphp: Slot [16] registered [ 0.273126] acpiphp: Slot [17] registered [ 0.276172] acpiphp: Slot [18] registered [ 0.277181] acpiphp: Slot [19] registered [ 0.280126] acpiphp: Slot [20] registered [ 0.281000] acpiphp: Slot [21] registered [ 0.282084] acpiphp: Slot [22] registered [ 0.283075] acpiphp: Slot [23] registered [ 0.285088] acpiphp: Slot [24] registered [ 0.286077] acpiphp: Slot [25] registered [ 0.287081] acpiphp: Slot [26] registered [ 0.289084] acpiphp: Slot [27] registered [ 0.290096] acpiphp: Slot [28] registered [ 0.292082] acpiphp: Slot [29] registered [ 0.293000] acpiphp: Slot [30] registered [ 0.293000] acpiphp: Slot [31] registered [ 0.294049] PCI host bridge to bus 0000:00 [ 0.296012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302013] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.303018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.305167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.308000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312804] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.326022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.329022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.339436] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342793] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344722] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.347014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.352840] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.356015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360632] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.366015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.372016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392064] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.404708] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.413015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.419017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.428019] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.432000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.437020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.442016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.481016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.490672] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.495014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.516025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.524567] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.530019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.542033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.584034] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.601744] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.637022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.647017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.654000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.656434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.658461] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.660378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.661247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.673646] iommu: Default domain type: Passthrough [ 0.675580] SCSI subsystem initialized [ 0.676152] ACPI: bus type USB registered [ 0.677000] usbcore: registered new interface driver usbfs [ 0.679092] usbcore: registered new interface driver hub [ 0.683161] usbcore: registered new device driver usb [ 0.684000] pps_core: LinuxPPS API ver. 1 registered [ 0.684000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.688071] PTP clock support registered [ 0.690100] EDAC MC: Ver: 3.0.0 [ 0.694155] PCI: Using ACPI for IRQ routing [ 0.695803] NetLabel: Initializing [ 0.701014] NetLabel: domain hash size = 128 [ 0.702000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.702000] NetLabel: unlabeled traffic allowed by default [ 0.708050] vgaarb: loaded [ 0.711299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.712008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727619] clocksource: Switched to clocksource kvm-clock [ 1.021820] VFS: Disk quotas dquot_6.6.0 [ 1.026355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.033692] *** VALIDATE ramfs *** [ 1.035682] *** VALIDATE hugetlbfs *** [ 1.037358] pnp: PnP ACPI init [ 1.041227] pnp: PnP ACPI: found 6 devices [ 1.067717] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.072922] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.076435] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.079714] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.083513] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.089370] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 1.094238] NET: Registered protocol family 2 [ 1.097491] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.107040] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.114489] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.122026] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.137394] TCP: Hash tables configured (established 65536 bind 65536) [ 1.140613] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.143141] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.145513] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.147921] NET: Registered protocol family 1 [ 1.151767] RPC: Registered named UNIX socket transport module. [ 1.153910] RPC: Registered udp transport module. [ 1.155143] RPC: Registered tcp transport module. [ 1.156286] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.159904] NET: Registered protocol family 44 [ 1.161174] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.162677] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.164290] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.166214] PCI: CLS 0 bytes, default 64 [ 1.167459] Unpacking initramfs... [ 3.623039] debug: unmapping init [mem 0xffff8aabbcc54000-0xffff8aabbffbffff] [ 3.632868] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.638320] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.644455] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.615168] Initialise system trusted keyrings [ 5.626700] Key type blacklist registered [ 5.634981] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.669691] zbud: loaded [ 5.685431] *** VALIDATE nfs *** [ 5.686694] *** VALIDATE nfs4 *** [ 5.688434] pstore: using deflate compression [ 5.693560] Platform Keyring initialized [ 5.914154] NET: Registered protocol family 38 [ 5.916312] Key type asymmetric registered [ 5.920635] Asymmetric key parser 'x509' registered [ 5.923216] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.930763] io scheduler mq-deadline registered [ 5.934149] io scheduler kyber registered [ 5.935515] io scheduler bfq registered [ 5.945153] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.952112] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.956457] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.963462] ACPI: Power Button [PWRF] [ 6.281127] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.475011] hrtimer: interrupt took 5990138 ns [ 6.640548] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.336123] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.586505] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.001444] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.133082] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.178919] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.184776] Non-volatile memory driver v1.3 [ 8.187451] Linux agpgart interface v0.103 [ 8.291205] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 8.320979] vda: detected capacity change from 0 to 68526080 [ 8.372148] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.382077] vdb: detected capacity change from 0 to 1073741824 [ 8.417399] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.427870] vdc: detected capacity change from 0 to 2621440000 [ 8.469176] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.479231] vdd: detected capacity change from 0 to 2621440000 [ 8.523713] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.532792] vde: detected capacity change from 0 to 4294967296 [ 8.561533] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.579731] vdf: detected capacity change from 0 to 4294967296 [ 8.598903] libphy: Fixed MDIO Bus: probed [ 8.612989] usbcore: registered new interface driver usbserial_generic [ 8.619996] usbserial: USB Serial support registered for generic [ 8.623775] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.629170] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.631242] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.633361] mousedev: PS/2 mouse device common for all mice [ 8.649961] rtc_cmos 00:05: RTC can wake from S4 [ 8.667385] rtc_cmos 00:05: registered as rtc0 [ 8.668911] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.669088] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.683141] intel_pstate: CPU model not supported [ 8.695390] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.708684] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.709056] hid: raw HID events driver (C) Jiri Kosina [ 8.717090] usbcore: registered new interface driver usbhid [ 8.720533] usbhid: USB HID core driver [ 8.721790] drop_monitor: Initializing network drop monitor service [ 8.736310] Initializing XFRM netlink socket [ 8.742751] NET: Registered protocol family 10 [ 8.745577] Segment Routing with IPv6 [ 8.748136] NET: Registered protocol family 17 [ 8.759308] mpls_gso: MPLS GSO support [ 8.770792] RAS: Correctable Errors collector initialized. [ 8.773484] AVX version of gcm_enc/dec engaged. [ 8.776552] AES CTR mode by8 optimization enabled [ 8.954421] sched_clock: Marking stable (8954401422, 0)->(10317369947, -1362968525) [ 8.960867] registered taskstats version 1 [ 8.965519] Loading compiled-in X.509 certificates [ 8.969361] zswap: loaded using pool lzo/zbud [ 9.041080] Key type big_key registered [ 9.084215] Key type encrypted registered [ 9.090771] ima: No TPM chip found, activating TPM-bypass! [ 9.092895] ima: Allocated hash algorithm: sha1 [ 9.098799] ima: No architecture policies found [ 9.100938] evm: Initialising EVM extended attributes: [ 9.103073] evm: security.selinux [ 9.104184] evm: security.ima [ 9.105209] evm: security.capability [ 9.106593] evm: HMAC attrs: 0x1 [ 9.110094] rtc_cmos 00:05: setting system clock to 2025-11-16 22:25:03 UTC (1763331903) [ 9.119833] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 9.130867] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 9.138133] Write protecting the kernel read-only data: 28672k [ 9.147103] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 9.149427] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 9.226354] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.246585] systemd[1]: Detected virtualization kvm. [ 9.248184] systemd[1]: Detected architecture x86-64. [ 9.249776] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.290309] systemd[1]: No hostname configured. [ 9.294149] systemd[1]: Set hostname to . [ 9.300228] random: systemd: uninitialized urandom read (16 bytes read) [ 9.302920] systemd[1]: Initializing machine ID from random generator. [ 9.583544] random: systemd: uninitialized urandom read (16 bytes read) [ 9.586143] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.590995] random: systemd: uninitialized urandom read (16 bytes read) [ 9.594650] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.602474] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.302619] device-mapper: uevent: version 1.0.3 [ 11.304812] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [0[ 13.374655] virtio_net virtio0 ens2: renamed from eth0 m] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.634217] random: fast init done [ 14.877240] scsi host0: ata_piix [ 15.179950] scsi host1: ata_piix [ 15.181099] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.183452] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.633030] random: crng init done [ 19.634591] random: 7 urandom warning(s) missed due to ratelimiting [ 21.032830] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.360660] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.203216] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.978692] SELinux: Disabled at runtime. [ 27.073143] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.096137] systemd[1]: Detected virtualization kvm. [ 27.098129] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.844823] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.851803] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.861838] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.873078] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.881702] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.908800] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.935728] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK [0[ 29.556058] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 30.574989] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.668346] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.859221] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.152127] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.813510] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 39.190879] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (10s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit)[ 41.089833] NFS: Registering the id_resolver key type [ 41.091887] Key type id_resolver registered [ 41.103801] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg455-server login: [ 98.425508] libcfs: loading out-of-tree module taints kernel. [ 98.438854] Key type ._llcrypt registered [ 98.440318] Key type .llcrypt registered [ 98.543583] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 112.461946] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 113.828970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 113.842213] alg: No test for adler32 (adler32-zlib) [ 115.191531] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 115.993584] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 117.704195] Key type lgssc registered [ 119.163656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.542748] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.295754] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.306029] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.953730] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 166.162728] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.826797] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 194.933584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.988034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 195.002041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.273181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.299984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.408718] Lustre: lustre-MDT0000: new disk, initializing [ 196.566081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.620862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 201.227510] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.736346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 214.860239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 214.970433] Lustre: 6524:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.043965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.057638] Lustre: Skipped 1 previous similar message [ 215.252801] Lustre: lustre-MDT0001: new disk, initializing [ 215.364526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.404272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.425443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 219.245347] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.101751] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.933218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 234.035577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 234.246721] Lustre: lustre-OST0000: new disk, initializing [ 234.264154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.339196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.298793] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.188304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.194188] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.256632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 255.252486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 255.373843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 255.499514] Lustre: lustre-OST0001: new disk, initializing [ 255.519329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.615686] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 261.682442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.685913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.783028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 262.081750] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.610781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.002228] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.312879] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.612183] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing yml_node [ 307.379538] Lustre: DEBUG MARKER: Client: 2.16.61.53 [ 310.325367] Lustre: DEBUG MARKER: MDS: 2.16.61.53 [ 313.868898] Lustre: DEBUG MARKER: OSS: 2.16.61.53 [ 315.899322] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:30:07 EST 2025 [ 336.044477] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 337.909729] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 353.761350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 353.763944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.792963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.798289] Lustre: Skipped 3 previous similar messages [ 358.848861] LustreError: 12232:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 358.989124] Lustre: server umount lustre-MDT0000 complete [ 363.228789] LustreError: 7459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332257 with bad export cookie 17953656294807102036 [ 363.247268] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.255923] LustreError: 7459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 364.001539] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.005267] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.005687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.046131] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 364.108184] Lustre: Skipped 4 previous similar messages [ 369.121000] LustreError: 9501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.121046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.150990] Lustre: Skipped 2 previous similar messages [ 369.534939] LustreError: 12435:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 369.537600] LustreError: 12435:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 369.693788] Lustre: server umount lustre-MDT0001 complete [ 373.785110] LustreError: 12637:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 373.793340] LustreError: 12637:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 373.844141] Lustre: server umount lustre-OST0000 complete [ 377.619537] LustreError: 12838:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 377.622674] LustreError: 12838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 377.769965] Lustre: server umount lustre-OST0001 complete [ 384.287653] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 394.170767] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 405.392000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 412.507900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 419.079304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 426.547847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 433.923282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 433.984201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.227487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.281554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.401847] Lustre: lustre-MDT0000: new disk, initializing [ 434.516374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.554230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 438.889700] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 449.985321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 450.099223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 450.214307] Lustre: 15023:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 450.277434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 450.281729] Lustre: Skipped 1 previous similar message [ 450.342454] Lustre: lustre-MDT0001: new disk, initializing [ 450.447324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.512892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.527975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 455.030235] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 463.361984] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 467.324606] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 474.746394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 474.826896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 475.036144] Lustre: lustre-OST0000: new disk, initializing [ 475.041317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 475.151097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.309806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.327771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.444434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 480.740925] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 488.812663] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.226647] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.451361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.125587] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.348281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 502.241388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 502.244810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 502.252164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.260610] Lustre: Skipped 1 previous similar message [ 506.507907] LustreError: 17964:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 506.530185] LustreError: 17964:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 506.724474] Lustre: server umount lustre-OST0000 complete [ 511.969946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 511.973417] LustreError: Skipped 1 previous similar message [ 511.990918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 512.010759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.019490] Lustre: Skipped 1 previous similar message [ 516.542489] LustreError: 18165:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 516.549461] LustreError: 18165:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 516.664543] Lustre: server umount lustre-MDT0000 complete [ 517.607567] LustreError: 17329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.628782] LustreError: 17329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 520.251031] LustreError: 15015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332414 with bad export cookie 17953656294807103569 [ 520.256889] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.275453] LustreError: 15015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 520.580691] Lustre: server umount lustre-MDT0001 complete [ 524.321067] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:33:36 (1763332416) [ 530.943154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.324107] LustreError: 18895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.388280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.070546] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 536.548377] LustreError: 18895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.676704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 547.605568] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 552.900936] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 556.361272] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 563.073539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 569.329547] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 575.050937] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.281352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 589.287850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.308453] Lustre: Skipped 1 previous similar message [ 589.317685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.327334] Lustre: Skipped 2 previous similar messages [ 592.516317] LustreError: 21142:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 592.539064] LustreError: 21142:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 592.626393] Lustre: server umount lustre-OST0000 complete [ 602.680582] Lustre: server umount lustre-MDT0000 complete [ 606.689834] LustreError: 18889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.712116] LustreError: 18889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 606.974798] LustreError: 18875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332501 with bad export cookie 17953656294807104451 [ 606.987224] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.996885] LustreError: 18875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 607.449865] Lustre: server umount lustre-MDT0001 complete [ 620.702360] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 624.083728] Key type lgssc unregistered [ 624.474633] LNet: 22080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.491455] LNetError: 22080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.510280] LNet: Removed LNI 192.168.204.155@tcp [ 625.632161] Key type .llcrypt unregistered [ 625.637310] Key type ._llcrypt unregistered [ 649.641178] Key type ._llcrypt registered [ 649.643524] Key type .llcrypt registered [ 649.739655] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 651.250543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 651.279678] alg: No test for adler32 (adler32-zlib) [ 652.501505] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 652.809097] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 654.584262] Key type lgssc registered [ 655.995466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 665.970122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 666.007839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.870363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.393116] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 678.370180] LustreError: 23473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.401144] LustreError: 23473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 681.563993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 681.956830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 687.042585] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 693.187760] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 698.820468] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 706.569515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 706.959384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.050890] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 713.160925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 718.557993] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 733.666342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.673504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.676323] LustreError: Skipped 1 previous similar message [ 733.681076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 733.708337] Lustre: Skipped 1 previous similar message [ 738.788299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.791403] Lustre: Skipped 2 previous similar messages [ 739.037151] LustreError: 26167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 739.208194] Lustre: server umount lustre-OST0000 complete [ 743.394267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 743.404664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 743.431169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.025296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 749.040695] Lustre: Skipped 2 previous similar messages [ 754.147134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.158752] Lustre: Skipped 1 previous similar message [ 757.216266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 757.332317] LustreError: 26368:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 757.339325] LustreError: 26368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 757.396884] Lustre: server umount lustre-MDT0000 complete [ 759.266270] LustreError: 23469:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.283885] LustreError: 23469:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 761.382465] LustreError: 26079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332655 with bad export cookie 10318695355547296379 [ 761.388431] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.430266] LustreError: 26079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 761.654251] LustreError: 26570:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 761.669254] LustreError: 26570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 761.942675] Lustre: server umount lustre-MDT0001 complete [ 772.411250] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 775.422562] Key type lgssc unregistered [ 775.701886] LNet: 27104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.715891] LNetError: 27104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.728873] LNet: Removed LNI 192.168.204.155@tcp [ 776.524354] Key type .llcrypt unregistered [ 776.533339] Key type ._llcrypt unregistered [ 796.576563] Key type ._llcrypt registered [ 796.578278] Key type .llcrypt registered [ 796.689091] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 797.460414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.548207] alg: No test for adler32 (adler32-zlib) [ 798.541055] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 798.727745] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 800.368191] Key type lgssc registered [ 801.252582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 808.126980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 808.141667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.587897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.194709] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 819.170637] LustreError: 28475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 819.194882] LustreError: 28475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 823.797471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 824.411327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 829.621845] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 835.450666] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 839.589927] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 852.242546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 852.541378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.580531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 857.508982] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 863.108823] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 873.957649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 873.957722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.048152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.059627] Lustre: Skipped 1 previous similar message [ 875.263826] LustreError: 31066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 875.455765] Lustre: server umount lustre-OST0000 complete [ 879.074666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.079353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.103017] Lustre: Skipped 2 previous similar messages [ 884.196797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.199342] Lustre: Skipped 2 previous similar messages [ 885.110304] LustreError: 31268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 885.123262] LustreError: 31268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 885.294745] Lustre: server umount lustre-MDT0000 complete [ 889.068482] LustreError: 28455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332783 with bad export cookie 6820311108012886983 [ 889.073960] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 889.108912] LustreError: 28455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 889.273633] LustreError: 31470:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 889.276598] LustreError: 31470:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 889.528447] Lustre: server umount lustre-MDT0001 complete [ 898.289881] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 900.917531] Key type lgssc unregistered [ 901.138168] LNet: 32002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.142765] LNetError: 32002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.154761] LNet: Removed LNI 192.168.204.155@tcp [ 901.950217] Key type .llcrypt unregistered [ 901.953500] Key type ._llcrypt unregistered [ 923.540053] Key type ._llcrypt registered [ 923.541312] Key type .llcrypt registered [ 923.642118] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 924.842594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.874104] alg: No test for adler32 (adler32-zlib) [ 925.984437] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 926.303741] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 928.016207] Key type lgssc registered [ 929.234716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.996181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 937.012737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.551148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 942.318658] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 947.685065] LustreError: 33394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.715382] LustreError: 33394:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 950.259313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.589866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.271299] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 959.481963] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.723514] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 968.182684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 968.427316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.693055] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 979.446463] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 982.772187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 991.202970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.212377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.228980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.252989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.252997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.253582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.269935] Lustre: Skipped 1 previous similar message [ 996.739483] LustreError: 35643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 996.882620] Lustre: server umount lustre-OST0000 complete [ 1001.954211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1001.960261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.979140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.296715] LustreError: 35845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1006.299969] LustreError: 35845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1006.386422] Lustre: server umount lustre-MDT0000 complete [ 1008.609205] LustreError: 33395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.629715] LustreError: 33395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1009.836961] LustreError: 33376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332904 with bad export cookie 1610019350077786390 [ 1009.841181] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.852716] LustreError: 33376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1010.018237] LustreError: 36047:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1010.021796] LustreError: 36047:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1010.244695] Lustre: server umount lustre-MDT0001 complete [ 1018.692803] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1021.292288] Key type lgssc unregistered [ 1021.593680] LNet: 36579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.613489] LNetError: 36579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.635706] LNet: Removed LNI 192.168.204.155@tcp [ 1022.406421] Key type .llcrypt unregistered [ 1022.411841] Key type ._llcrypt unregistered [ 1043.966304] Key type ._llcrypt registered [ 1043.967975] Key type .llcrypt registered [ 1044.071921] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1044.763170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1044.802033] alg: No test for adler32 (adler32-zlib) [ 1045.852610] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1046.085254] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1047.811157] Key type lgssc registered [ 1049.143159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1056.975855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1057.018455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.534293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.266897] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1063.909924] LustreError: 37974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.918350] LustreError: 37974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1068.008406] LustreError: 37974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.550699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1072.988102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1078.679422] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1084.578599] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1088.010587] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1096.063455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1096.502554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.171228] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1102.653587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1108.026916] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1115.684622] Lustre: lustre-OST0000: Not available for connect from 192.168.204.55@tcp (stopping) [ 1118.177498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1118.185858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.193143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1118.195355] Lustre: Skipped 1 previous similar message [ 1120.395439] LustreError: 40221:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1120.518617] Lustre: server umount lustre-OST0000 complete [ 1142.754240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.759521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.776161] Lustre: Skipped 1 previous similar message [ 1142.778776] Lustre: Skipped 1 previous similar message [ 1147.880464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.887538] Lustre: Skipped 1 previous similar message [ 1152.992822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.007651] Lustre: Skipped 1 previous similar message [ 1154.016253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1154.122056] LustreError: 40596:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1154.144651] LustreError: 40596:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1154.262650] Lustre: server umount lustre-MDT0000 complete [ 1158.121682] LustreError: 37974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.136179] LustreError: 37974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1158.168905] LustreError: 37953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333052 with bad export cookie 4248209129665225084 [ 1158.181196] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.191068] LustreError: 37953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1164.500523] LustreError: 40799:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1164.513761] LustreError: 40799:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1164.722742] Lustre: server umount lustre-MDT0001 complete [ 1175.301765] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1179.342952] Key type lgssc unregistered [ 1179.729826] LNet: 41333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.755483] LNetError: 41333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.775850] LNet: Removed LNI 192.168.204.155@tcp [ 1182.001744] Key type .llcrypt unregistered [ 1182.003097] Key type ._llcrypt unregistered [ 1207.951747] Key type ._llcrypt registered [ 1207.953461] Key type .llcrypt registered [ 1208.067585] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1208.780358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1208.852759] alg: No test for adler32 (adler32-zlib) [ 1209.944174] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1210.273273] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1211.992310] Key type lgssc registered [ 1213.134775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1220.393527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1220.404780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.895731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.714695] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1232.362409] LustreError: 42729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.384635] LustreError: 42729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1233.930951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1234.340559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1238.356240] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1244.178201] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1247.811542] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1254.838590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1255.163195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.631559] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1266.272188] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1267.508149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1273.224546] LustreError: 44979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1273.375647] Lustre: server umount lustre-MDT0000 complete [ 1274.398583] LustreError: 42723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.361658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.367762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.929546] LustreError: 43448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333171 with bad export cookie 16912647708397214218 [ 1276.938982] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.949901] LustreError: 43448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1277.922579] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.931174] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1277.936698] Lustre: Skipped 1 previous similar message [ 1277.942925] LustreError: 42724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.985682] LustreError: 42724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1279.525762] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.55@tcp (stopping) [ 1281.953199] LustreError: 42724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.958796] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1281.988893] LustreError: 42724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1283.191905] LustreError: 45182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1283.197910] LustreError: 45182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1283.330850] Lustre: server umount lustre-MDT0001 complete [ 1303.682973] LustreError: 45405:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1303.690487] LustreError: 45405:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1303.831419] Lustre: server umount lustre-OST0000 complete [ 1318.918408] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1321.565965] Key type lgssc unregistered [ 1321.878915] LNet: 46244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.886029] LNetError: 46244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.902884] LNet: Removed LNI 192.168.204.155@tcp [ 1322.794821] Key type .llcrypt unregistered [ 1322.796327] Key type ._llcrypt unregistered [ 1345.295776] Key type ._llcrypt registered [ 1345.297322] Key type .llcrypt registered [ 1345.434849] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1346.641531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1346.654104] alg: No test for adler32 (adler32-zlib) [ 1347.766822] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1347.980701] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1349.672189] Key type lgssc registered [ 1350.808104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1359.156692] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1359.216948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1374.816313] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1379.938067] LustreError: 47610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 1406.689558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.429620] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1417.682288] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1583.133915] LustreError: 48319:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1583.223927] Lustre: server umount lustre-OST0000 complete [ 1594.676724] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1596.877683] Key type lgssc unregistered [ 1597.155602] LNet: 49155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.171929] LNetError: 49155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.187992] LNet: Removed LNI 192.168.204.155@tcp [ 1597.765775] Key type .llcrypt unregistered [ 1597.767420] Key type ._llcrypt unregistered [ 1616.756168] Key type ._llcrypt registered [ 1616.757974] Key type .llcrypt registered [ 1616.851767] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1617.751357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.763836] alg: No test for adler32 (adler32-zlib) [ 1618.852368] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1619.073516] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1620.720122] Key type lgssc registered [ 1621.809730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1628.688249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1628.715383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.207842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.371268] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1640.418491] LustreError: 50527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.449646] LustreError: 50527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1641.724830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1642.160116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1646.083992] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1651.448642] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1655.102759] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1661.943247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1662.345305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.780067] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1672.814991] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1675.874389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.881732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1680.957268] LustreError: 52772:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1681.136748] Lustre: server umount lustre-OST0000 complete [ 1686.497827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.503206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.529205] Lustre: Skipped 1 previous similar message [ 1686.547610] Lustre: Skipped 2 previous similar messages [ 1691.617322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1691.627810] Lustre: Skipped 1 previous similar message [ 1695.718281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.296126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1699.373669] LustreError: 52973:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1699.412747] LustreError: 52973:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1699.549330] Lustre: server umount lustre-MDT0000 complete [ 1700.833962] LustreError: 50527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.844048] LustreError: 50527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1702.809757] LustreError: 50509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333597 with bad export cookie 18237795644411082121 [ 1702.815616] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1702.933631] LustreError: 53177:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1702.943599] LustreError: 53177:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1703.099274] Lustre: server umount lustre-MDT0001 complete [ 1712.594922] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1715.352390] Key type lgssc unregistered [ 1715.688685] LNet: 53709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.706544] LNetError: 53709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1715.733778] LNet: Removed LNI 192.168.204.155@tcp [ 1716.491496] Key type .llcrypt unregistered [ 1716.495148] Key type ._llcrypt unregistered [ 1734.892158] Key type ._llcrypt registered [ 1734.893790] Key type .llcrypt registered [ 1734.977264] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1735.710769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1735.804642] alg: No test for adler32 (adler32-zlib) [ 1736.920493] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1737.162684] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1738.833762] Key type lgssc registered [ 1740.144441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.133802] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1748.163552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1763.811329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1768.928211] LustreError: 55053:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 1795.616165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.323520] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1805.197506] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1811.376693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.768908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.848155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1815.351772] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1817.061767] LustreError: 55079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.069520] LustreError: 55079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1822.184470] LustreError: 55079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.651426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1824.133169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.720752] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1833.886781] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.520750] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1839.589112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.608853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.648594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.639675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.646764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1845.575221] LustreError: 57332:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1845.925510] Lustre: server umount lustre-OST0000 complete [ 1867.236333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.240416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.266427] Lustre: Skipped 1 previous similar message [ 1867.270350] Lustre: Skipped 2 previous similar messages [ 1872.356918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.360902] Lustre: Skipped 1 previous similar message [ 1872.420141] LustreError: 57710:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1872.423490] LustreError: 57710:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1872.510560] Lustre: server umount lustre-MDT0000 complete [ 1875.736891] LustreError: 56705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333770 with bad export cookie 3811476524175625410 [ 1875.743387] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1890.272150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1890.376312] LustreError: 57912:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1890.389187] LustreError: 57912:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1890.526244] Lustre: server umount lustre-MDT0001 complete [ 1898.304786] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 1900.434815] Key type lgssc unregistered [ 1900.709516] LNet: 58444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.715757] LNetError: 58444:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.723693] LNet: Removed LNI 192.168.204.155@tcp [ 1901.288074] Key type .llcrypt unregistered [ 1901.289763] Key type ._llcrypt unregistered [ 1918.507352] Key type ._llcrypt registered [ 1918.509199] Key type .llcrypt registered [ 1918.590906] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 1919.100266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.157391] alg: No test for adler32 (adler32-zlib) [ 1920.173644] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 1920.322324] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 1921.952828] Key type lgssc registered [ 1922.986636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.564225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.588822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.973106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.402712] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1941.473525] LustreError: 59816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.490181] LustreError: 59816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1942.540879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1942.946713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1946.846946] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1952.916438] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.186796] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1961.851903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1962.289899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.286372] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 1975.336769] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1976.640892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1994.224862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.229664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.240924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1994.887546] LustreError: 62085:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1995.072330] Lustre: server umount lustre-OST0000 complete [ 1999.329787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1999.338559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.351562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.403565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.405878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2002.452307] Lustre: Skipped 1 previous similar message [ 2007.521846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.533520] Lustre: Skipped 1 previous similar message [ 2009.572690] LustreError: 62286:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2009.579765] LustreError: 62286:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2009.665516] Lustre: server umount lustre-MDT0000 complete [ 2011.629989] LustreError: 59816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.656736] LustreError: 59816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2012.957336] LustreError: 59795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333907 with bad export cookie 114942845793767791 [ 2012.964476] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2012.979597] LustreError: 59795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2013.146964] LustreError: 62489:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2013.186609] LustreError: 62489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2013.404871] Lustre: server umount lustre-MDT0001 complete [ 2022.009534] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2024.689082] Key type lgssc unregistered [ 2025.014703] LNet: 63022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2025.021626] LNetError: 63022:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.037185] LNet: Removed LNI 192.168.204.155@tcp [ 2025.855623] Key type .llcrypt unregistered [ 2025.856989] Key type ._llcrypt unregistered [ 2055.591989] Key type ._llcrypt registered [ 2055.593774] Key type .llcrypt registered [ 2055.696538] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2056.538607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.559102] alg: No test for adler32 (adler32-zlib) [ 2057.611432] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2057.920931] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2059.602192] Key type lgssc registered [ 2060.521174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.749423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.760958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.154373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.230477] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2077.734053] LustreError: 64710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.743977] LustreError: 64710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2078.566287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2078.994681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2082.444988] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2087.000295] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.928725] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.341356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2094.631748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.126733] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2104.781369] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2113.113397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2121.516496] Lustre: Failing over lustre-MDT0000 [ 2121.693198] LustreError: 66975:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2121.829968] Lustre: server umount lustre-MDT0000 complete [ 2122.263747] LustreError: 64707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.235079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.256139] Lustre: Skipped 2 previous similar messages [ 2127.385675] LustreError: 64707:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.402618] LustreError: 64707:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2130.991663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.087312] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.211418] Lustre: *** cfs_fail_loc=135, val=0*** [ 2131.347347] LustreError: 67487:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2131.356921] LustreError: 67487:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.155@tcp: cfg command failed: rc = -2 [ 2131.368208] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2131.368208] [ 2131.374353] LustreError: MGC192.168.204.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2131.398088] LustreError: 67475:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2131.420504] LustreError: 67475:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2131.429341] LustreError: 67475:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2131.464181] Lustre: server umount lustre-MDT0000 complete [ 2131.466479] LustreError: 67475:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2132.482979] LustreError: 64705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.490098] LustreError: 64705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2137.600161] LustreError: 66676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.629598] LustreError: 66676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2138.090397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.234521] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.636776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.718824] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2142.606272] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2142.721903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2143.732348] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2143.759116] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2143.796096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2149.858121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.860766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.876732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2151.822900] LustreError: 68295:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2151.827932] LustreError: 68295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2151.948453] Lustre: server umount lustre-OST0000 complete [ 2159.074528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.078782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.106386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.275087] Lustre: Skipped 1 previous similar message [ 2159.329583] Lustre: Skipped 1 previous similar message [ 2164.195442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.207781] Lustre: Skipped 1 previous similar message [ 2165.234988] LustreError: 68497:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2165.252795] LustreError: 68497:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2165.363287] Lustre: server umount lustre-MDT0000 complete [ 2169.318643] LustreError: 64705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.338581] LustreError: 64705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2170.003636] LustreError: 64691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334064 with bad export cookie 17504983182038697754 [ 2170.010869] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.031191] LustreError: 64691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2170.314538] LustreError: 68712:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2170.335297] LustreError: 68712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2170.647805] Lustre: server umount lustre-MDT0001 complete [ 2182.918888] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2185.616661] Key type lgssc unregistered [ 2185.859137] LNet: 69244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.865555] LNetError: 69244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.879475] LNet: Removed LNI 192.168.204.155@tcp [ 2186.657356] Key type .llcrypt unregistered [ 2186.659279] Key type ._llcrypt unregistered [ 2204.936632] Key type ._llcrypt registered [ 2204.940491] Key type .llcrypt registered [ 2205.028652] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2205.573742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.658743] alg: No test for adler32 (adler32-zlib) [ 2206.702510] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2206.912484] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2208.568137] Key type lgssc registered [ 2209.411740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2215.620881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2215.646192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.137261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2220.899622] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2226.660832] LustreError: 70616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.676286] LustreError: 70616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2228.875340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2229.281289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2232.642838] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2237.739760] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2241.301317] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2247.056958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2247.405343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.294475] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2257.738744] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2261.756284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2268.380763] Lustre: Failing over lustre-MDT0000 [ 2268.530690] LustreError: 72862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2268.622885] Lustre: server umount lustre-MDT0000 complete [ 2269.736631] LustreError: 70611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.178580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2270.188681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.226204] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.234878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.251873] LustreError: 70617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2274.822878] LustreError: 70611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.861383] LustreError: 70611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2277.193940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.319330] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.444333] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2277.566382] LustreError: 73370:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2277.573373] LustreError: 73370:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.155@tcp: cfg command failed: rc = -9 [ 2277.590220] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2277.590220] [ 2277.596923] LustreError: MGC192.168.204.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2277.612917] LustreError: 73358:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2277.618789] LustreError: 73358:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2277.623914] LustreError: 73358:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2277.655300] Lustre: server umount lustre-MDT0000 complete [ 2277.656815] LustreError: 73358:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2279.948216] LustreError: 70613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.991510] LustreError: 70613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2284.015262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.089698] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.103517] LustreError: 70608:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2284.326110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.418188] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2285.974578] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2287.924470] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2289.649668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2289.684460] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2289.720718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2294.776540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.784144] Lustre: Skipped 2 previous similar messages [ 2294.788132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.790649] Lustre: Skipped 1 previous similar message [ 2296.697298] LustreError: 74185:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2296.707532] LustreError: 74185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2296.836829] Lustre: server umount lustre-OST0000 complete [ 2299.878860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2299.882141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2299.904274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.993496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.996877] Lustre: Skipped 2 previous similar messages [ 2305.744396] LustreError: 74386:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2305.746739] LustreError: 74386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2305.853402] Lustre: server umount lustre-MDT0000 complete [ 2308.286786] LustreError: 70597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334202 with bad export cookie 8116209005856287257 [ 2308.297143] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.401337] LustreError: 74587:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2308.405239] LustreError: 74587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2308.527763] Lustre: server umount lustre-MDT0001 complete [ 2314.498334] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2316.217763] Key type lgssc unregistered [ 2316.435266] LNet: 75118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.439634] LNetError: 75118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.480320] LNet: Removed LNI 192.168.204.155@tcp [ 2318.022940] Key type .llcrypt unregistered [ 2318.025202] Key type ._llcrypt unregistered [ 2327.158531] Key type ._llcrypt registered [ 2327.160417] Key type .llcrypt registered [ 2327.257979] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2327.478285] Key type .llcrypt unregistered [ 2327.481299] Key type ._llcrypt unregistered [ 2344.958659] Key type ._llcrypt registered [ 2344.960231] Key type .llcrypt registered [ 2345.026975] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2345.735652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2345.837188] alg: No test for adler32 (adler32-zlib) [ 2346.830654] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2346.974718] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2348.632284] Key type lgssc registered [ 2349.534821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.398348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.430547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.883809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.385404] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2368.482678] LustreError: 76960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.495173] LustreError: 76960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2368.912667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2369.218689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2372.896220] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2377.981631] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2381.007243] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2386.622908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2386.868422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.295866] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2396.622554] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2403.254430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2413.542348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.542375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.546955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.548408] Lustre: Skipped 1 previous similar message [ 2417.213420] LustreError: 79228:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2417.311254] Lustre: server umount lustre-OST0000 complete [ 2420.707950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2420.711491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.722567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.732106] Lustre: Skipped 1 previous similar message [ 2423.777161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.782682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.223356] LustreError: 79429:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2426.228460] LustreError: 79429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2426.344205] Lustre: server umount lustre-MDT0000 complete [ 2428.903916] LustreError: 79117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.930089] LustreError: 79117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2429.842596] LustreError: 78577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334324 with bad export cookie 9158667154490613013 [ 2429.848086] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.856836] LustreError: 78577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2430.084321] LustreError: 79631:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2430.110119] LustreError: 79631:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2430.316425] Lustre: server umount lustre-MDT0001 complete [ 2438.720940] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2440.889394] Key type lgssc unregistered [ 2441.150375] LNet: 80164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.156760] LNetError: 80164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.174870] LNet: Removed LNI 192.168.204.155@tcp [ 2441.830770] Key type .llcrypt unregistered [ 2441.832686] Key type ._llcrypt unregistered [ 2461.741463] Key type ._llcrypt registered [ 2461.743066] Key type .llcrypt registered [ 2461.941762] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2463.671823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.689361] alg: No test for adler32 (adler32-zlib) [ 2464.814385] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2465.059734] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2466.744193] Key type lgssc registered [ 2467.940508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.114426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.144598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.602179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.266438] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2485.731899] LustreError: 81539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.758866] LustreError: 81539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2487.914493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2488.260363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.104586] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2496.696751] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2500.041228] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2506.900570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2507.305049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2513.203734] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2517.999207] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2520.699887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2528.740652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.744234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.765544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.302828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2531.304513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.325413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.952107] LustreError: 83785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2534.084095] Lustre: server umount lustre-OST0000 complete [ 2539.489308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2539.494960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.510591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.526727] Lustre: Skipped 1 previous similar message [ 2541.538443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2541.540310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2541.561369] Lustre: Skipped 1 previous similar message [ 2543.568821] LustreError: 83986:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2543.578681] LustreError: 83986:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2543.716178] Lustre: server umount lustre-MDT0000 complete [ 2546.658210] LustreError: 81538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2546.679672] LustreError: 81538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2546.979148] LustreError: 83987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334441 with bad export cookie 5315366080155497292 [ 2546.989053] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.999738] LustreError: 83987:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2547.097300] LustreError: 84190:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2547.104604] LustreError: 84190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2547.251914] Lustre: server umount lustre-MDT0001 complete [ 2554.730987] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2557.101734] Key type lgssc unregistered [ 2557.325775] LNet: 84722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.334373] LNetError: 84722:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2558.379192] LNet: Removed LNI 192.168.204.155@tcp [ 2559.154981] Key type .llcrypt unregistered [ 2559.158099] Key type ._llcrypt unregistered [ 2575.957267] Key type ._llcrypt registered [ 2575.965048] Key type .llcrypt registered [ 2576.085299] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2577.045303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2577.104694] alg: No test for adler32 (adler32-zlib) [ 2578.177859] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2578.475665] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2580.184209] Key type lgssc registered [ 2581.197948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2588.107745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2588.122394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.477627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2592.628973] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2598.884450] LustreError: 86094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.915870] LustreError: 86094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2599.580643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2599.885521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2602.972528] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2606.788754] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2609.818421] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2614.236585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2614.440167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.774103] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2621.630830] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.910426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2645.476540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2645.497067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.526339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2648.041867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2648.051862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.078151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2649.577142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2649.587013] Lustre: Skipped 1 previous similar message [ 2651.396109] LustreError: 88357:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2651.608468] Lustre: server umount lustre-OST0000 complete [ 2656.227325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2656.230643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.240194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.809904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.274110] LustreError: 88561:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2661.297929] LustreError: 88561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2661.373913] Lustre: server umount lustre-MDT0000 complete [ 2664.645699] LustreError: 86072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334559 with bad export cookie 430003520515714411 [ 2664.648501] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.663892] LustreError: 86072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2664.842120] LustreError: 88762:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2664.857779] LustreError: 88762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2665.155208] Lustre: server umount lustre-MDT0001 complete [ 2673.704304] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2676.117664] Key type lgssc unregistered [ 2676.413450] LNet: 89296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2676.429484] LNetError: 89296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.486050] LNet: Removed LNI 192.168.204.155@tcp [ 2678.209935] Key type .llcrypt unregistered [ 2678.212946] Key type ._llcrypt unregistered [ 2697.541368] Key type ._llcrypt registered [ 2697.542526] Key type .llcrypt registered [ 2697.674444] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2698.341397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2698.390822] alg: No test for adler32 (adler32-zlib) [ 2699.416866] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2699.614546] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2701.248211] Key type lgssc registered [ 2702.418726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2709.903543] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2709.917698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2725.472434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2730.592297] LustreError: 90643:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 2757.216601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.871753] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2766.506600] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2771.673216] LustreError: 91501:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2771.782164] Lustre: server umount lustre-OST0000 complete [ 2779.708972] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:11:11 (1763334671) [ 2785.860155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.272914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.737763] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2791.399313] LustreError: 92172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.410385] LustreError: 92172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2796.522599] LustreError: 92172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.156818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2797.567268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.084723] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2807.648169] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2811.362392] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2818.942419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2819.407873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.001807] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2830.712747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2831.381194] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.067670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2841.078538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.083635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2841.090462] Lustre: Skipped 1 previous similar message [ 2846.185858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.194251] Lustre: Skipped 2 previous similar messages [ 2846.539215] LustreError: 94576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2846.541737] LustreError: 94576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2846.643841] Lustre: server umount lustre-OST0000 complete [ 2851.302584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.308673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2851.318226] Lustre: Skipped 1 previous similar message [ 2851.325580] Lustre: Skipped 1 previous similar message [ 2856.418616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2856.427890] Lustre: Skipped 1 previous similar message [ 2856.498967] LustreError: 94777:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2856.501705] LustreError: 94777:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2856.665185] Lustre: server umount lustre-MDT0000 complete [ 2860.415387] LustreError: 92153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334754 with bad export cookie 9898261310109020660 [ 2860.426611] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.592295] LustreError: 94979:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2860.606873] LustreError: 94979:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2860.826946] Lustre: server umount lustre-MDT0001 complete [ 2870.015863] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2872.531443] Key type lgssc unregistered [ 2872.863287] LNet: 95512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2872.879604] LNetError: 95512:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.895719] LNet: Removed LNI 192.168.204.155@tcp [ 2873.707372] Key type .llcrypt unregistered [ 2873.712701] Key type ._llcrypt unregistered [ 2892.925776] Key type ._llcrypt registered [ 2892.927457] Key type .llcrypt registered [ 2893.021400] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 2893.620434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2893.728783] alg: No test for adler32 (adler32-zlib) [ 2894.759168] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 2894.929235] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 2896.648189] Key type lgssc registered [ 2897.540571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2904.349756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2904.379842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.899989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.097470] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2916.095634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2916.393793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.537750] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2925.362892] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.310567] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2935.846730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2936.259836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2941.850441] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 2947.926352] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2951.675670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2961.897953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.903697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.906447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.933759] Lustre: Skipped 1 previous similar message [ 2964.873355] LustreError: 99134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2964.993990] Lustre: server umount lustre-OST0000 complete [ 2972.131532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.138958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.141810] Lustre: Skipped 1 previous similar message [ 2974.612553] LustreError: 99336:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2974.619589] LustreError: 99336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2974.819815] Lustre: server umount lustre-MDT0000 complete [ 2977.250542] LustreError: 96886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.279124] LustreError: 96886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2978.524600] LustreError: 96868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334872 with bad export cookie 11890676282369253569 [ 2978.532522] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2978.667057] LustreError: 99540:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2978.669740] LustreError: 99540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2978.873512] Lustre: server umount lustre-MDT0001 complete [ 2988.718237] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 2991.387873] Key type lgssc unregistered [ 2991.613340] LNet: 100073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2991.620335] LNetError: 100073:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2992.688805] LNet: Removed LNI 192.168.204.155@tcp [ 2993.409403] Key type .llcrypt unregistered [ 2993.411223] Key type ._llcrypt unregistered [ 3009.527702] Key type ._llcrypt registered [ 3009.534917] Key type .llcrypt registered [ 3009.624495] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3010.629930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.688477] alg: No test for adler32 (adler32-zlib) [ 3011.781615] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3012.143726] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 3013.904203] Key type lgssc registered [ 3015.192697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3023.275476] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3023.308414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3038.944330] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3044.065277] LustreError: 101293:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 3070.818493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.745238] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3082.162705] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3088.752420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.052322] LustreError: MGC192.168.204.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3089.062157] LustreError: 102205:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3089.069535] LustreError: 102205:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3089.076916] LustreError: 102205:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3089.153279] Lustre: server umount lustre-MDT0000 complete [ 3089.155187] LustreError: 102205:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3104.739475] LustreError: 102636:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3104.848231] Lustre: server umount lustre-OST0000 complete [ 3113.367590] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3121.004943] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3129.074959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3135.131089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.936772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3147.594505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3154.718602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3154.802394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3155.031670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3155.058096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3155.112966] Lustre: lustre-MDT0000: new disk, initializing [ 3155.182339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.199544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3158.742349] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3170.073431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3170.138396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3170.237155] Lustre: 104972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3170.299397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3170.301913] Lustre: Skipped 1 previous similar message [ 3170.393716] Lustre: lustre-MDT0001: new disk, initializing [ 3170.522714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3170.563782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3170.584036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3174.545885] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3181.722761] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3185.185680] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3191.498772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.563386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3191.791494] Lustre: lustre-OST0000: new disk, initializing [ 3191.794188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3191.910724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.530790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3197.544310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3197.791902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3197.844786] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3204.880659] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3208.951946] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3209.198155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3213.313533] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3213.508893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3218.404636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3218.411687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.412151] LustreError: Skipped 1 previous similar message [ 3218.425486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3218.430217] Lustre: Skipped 1 previous similar message [ 3221.574495] LustreError: 107916:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3221.585237] LustreError: 107916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3221.767184] Lustre: server umount lustre-OST0000 complete [ 3227.112408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3227.140585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.168067] Lustre: Skipped 1 previous similar message [ 3227.172797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.649649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.652962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.737931] Lustre: Skipped 1 previous similar message [ 3231.813116] LustreError: 108117:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3231.819955] LustreError: 108117:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3231.918412] Lustre: server umount lustre-MDT0000 complete [ 3235.993267] LustreError: 106723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335130 with bad export cookie 8245111102611188587 [ 3235.999568] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3236.017593] LustreError: 106723:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3236.224334] LustreError: 108319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3236.247252] LustreError: 108319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3236.592742] Lustre: server umount lustre-MDT0001 complete [ 3245.623248] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:18:57 (1763335137) [ 3249.569361] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3266.715608] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3275.790196] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3284.415967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3291.518615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3298.035092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3304.294430] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3310.903555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3310.970564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.172223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3311.235373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3311.307577] Lustre: lustre-MDT0000: new disk, initializing [ 3311.435962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.472405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3315.090682] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3324.978560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3325.043449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.106383] Lustre: 111454:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3325.150882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3325.160743] Lustre: Skipped 1 previous similar message [ 3325.250441] Lustre: lustre-MDT0001: new disk, initializing [ 3325.348883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.383090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3325.393143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3329.382615] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3336.473977] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3340.305781] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3347.206471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3347.285353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3347.569414] Lustre: lustre-OST0000: new disk, initializing [ 3347.577982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3347.702639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.237447] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3354.251627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3354.278742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3354.434794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3360.718890] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3364.724671] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3364.934609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3369.322962] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3369.526926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.073654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.079081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.095438] LustreError: Skipped 1 previous similar message [ 3375.115975] Lustre: Skipped 1 previous similar message [ 3375.118214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.671126] LustreError: 114396:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3377.676787] LustreError: 114396:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3377.867286] Lustre: server umount lustre-OST0000 complete [ 3381.729680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3381.744881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.767993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.770813] Lustre: Skipped 1 previous similar message [ 3387.756154] LustreError: 114600:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3387.758086] LustreError: 114600:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3387.947196] Lustre: server umount lustre-MDT0000 complete [ 3390.435508] LustreError: 111466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.461100] LustreError: 111466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3391.577780] LustreError: 111446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335285 with bad export cookie 8245111102611189469 [ 3391.580327] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3391.589963] LustreError: 111446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3392.042103] Lustre: server umount lustre-MDT0001 complete [ 3400.717709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.265238] LustreError: 115279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.380355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.125845] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3406.823209] LustreError: 115279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.924361] LustreError: 115279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.778880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3417.609297] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3423.041130] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3426.556798] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3432.186821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3437.500086] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3442.926616] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3454.214219] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3459.554622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.566747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.593709] Lustre: Skipped 1 previous similar message [ 3459.602041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3459.604475] Lustre: Skipped 2 previous similar messages [ 3463.748503] LustreError: 117622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3463.751588] LustreError: 117622:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3463.955640] Lustre: server umount lustre-OST0000 complete [ 3470.305170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.309107] LustreError: Skipped 1 previous similar message [ 3473.921053] Lustre: server umount lustre-MDT0000 complete [ 3476.964100] LustreError: 115279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.000030] LustreError: 115279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3477.779247] LustreError: 115259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335372 with bad export cookie 8245111102611190351 [ 3477.785140] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3477.788926] LustreError: 115259:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3478.267761] Lustre: server umount lustre-MDT0001 complete [ 3489.800408] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 3492.764702] Key type lgssc unregistered [ 3493.093518] LNet: 118557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3493.107628] LNetError: 118557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3493.124828] LNet: Removed LNI 192.168.204.155@tcp [ 3493.851110] Key type .llcrypt unregistered [ 3493.856265] Key type ._llcrypt unregistered [ 3509.288750] Key type ._llcrypt registered [ 3509.290037] Key type .llcrypt registered [ 3509.381441] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 3521.299352] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 3522.206624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3522.231546] alg: No test for adler32 (adler32-zlib) [ 3523.255958] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 3523.513000] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 3525.168187] Key type lgssc registered [ 3526.133204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3533.970979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.173635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.834164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.503289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.625480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.671844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3556.684617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.870597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.903141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3557.952571] Lustre: lustre-MDT0000: new disk, initializing [ 3558.028765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.052868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3561.136857] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3571.020682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3571.148813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3571.355388] Lustre: 121662:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3571.420529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3571.426872] Lustre: Skipped 1 previous similar message [ 3571.546247] Lustre: lustre-MDT0001: new disk, initializing [ 3571.717629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3571.808225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3571.818404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3575.253295] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3582.272447] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.329558] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3590.954079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3591.037909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.346544] Lustre: lustre-OST0000: new disk, initializing [ 3591.356606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.451558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3596.910736] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3599.449302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3599.476763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3599.593303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3603.868663] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3607.879433] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3608.087963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3611.540491] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3611.689905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3615.203383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3615.212926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3615.234707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.521840] LustreError: 124600:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3619.655939] Lustre: server umount lustre-OST0000 complete [ 3625.442377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.452430] Lustre: Skipped 1 previous similar message [ 3625.461475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.464017] Lustre: Skipped 1 previous similar message [ 3629.317631] LustreError: 124802:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3629.320734] LustreError: 124802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3629.414298] Lustre: server umount lustre-MDT0000 complete [ 3632.271226] LustreError: 121651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335526 with bad export cookie 11121898373030551161 [ 3632.274146] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3632.284357] LustreError: 121651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3632.384369] LustreError: 125003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3632.401000] LustreError: 125003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3632.511844] Lustre: server umount lustre-MDT0001 complete [ 3639.977629] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:25:32 (1763335532) [ 3646.639099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.312321] LustreError: 125676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.333857] LustreError: 125676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3647.441816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3651.546261] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3652.578049] LustreError: 125676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.676318] LustreError: 125676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3659.847523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3660.353753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3664.078595] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3668.693884] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3671.862891] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3675.617390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3675.621084] LustreError: Skipped 1 previous similar message [ 3675.625652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.636505] Lustre: Skipped 1 previous similar message [ 3675.644293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.667370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.668300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.701443] Lustre: Skipped 1 previous similar message [ 3679.924328] LustreError: 127019:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3679.927925] LustreError: 127019:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3680.034864] Lustre: server umount lustre-MDT0000 complete [ 3682.788362] LustreError: 125672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3682.805809] LustreError: 125672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3683.173702] LustreError: 125657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335577 with bad export cookie 11121898373030552043 [ 3683.183387] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3683.185700] LustreError: 125657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3683.501621] Lustre: server umount lustre-MDT0001 complete [ 3691.532361] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:26:23 (1763335583) [ 3697.539193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3713.057627] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3718.176502] LustreError: 127867:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 3743.904297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.093620] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3754.626908] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3756.514030] LustreError: 128532:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3756.516833] LustreError: 128532:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3756.690791] Lustre: server umount lustre-OST0000 complete [ 3764.699422] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:27:36 (1763335656) [ 3770.680777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3771.182621] LustreError: 129203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3771.244733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.663641] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3781.604299] LustreError: 129202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.611102] LustreError: 129202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3783.208118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.939555] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3791.801519] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3795.290782] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3800.772407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3806.262451] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3810.671669] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3821.546037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.548622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.556662] Lustre: Skipped 2 previous similar messages [ 3821.561015] Lustre: Skipped 1 previous similar message [ 3826.533934] LustreError: 131450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3826.542494] LustreError: 131450:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3826.655316] Lustre: server umount lustre-MDT0000 complete [ 3826.677063] LustreError: 131362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.693947] LustreError: 131362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3829.796570] LustreError: 129184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335724 with bad export cookie 11121898373030552708 [ 3829.797736] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3829.809388] LustreError: 129184:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3830.032661] Lustre: server umount lustre-MDT0001 complete [ 3843.134715] LustreError: 131854:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3843.138849] LustreError: 131854:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3843.273983] Lustre: server umount lustre-OST0000 complete [ 3850.101502] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:29:02 (1763335742) [ 3855.747578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.119654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.122606] Lustre: Skipped 2 previous similar messages [ 3858.933463] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3861.473230] LustreError: 132525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.481701] LustreError: 132525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3865.336635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3868.049485] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3871.763115] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3874.792979] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3880.459474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3881.724311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3885.681183] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3891.996197] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3896.265272] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3896.442306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3900.002496] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3900.159908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3902.438096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3902.441500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.460078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3902.464377] Lustre: Skipped 1 previous similar message [ 3905.519631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3907.766532] LustreError: 135272:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3907.775386] LustreError: 135272:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3907.903555] Lustre: server umount lustre-OST0000 complete [ 3911.648983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3917.280555] Lustre: server umount lustre-MDT0000 complete [ 3919.981765] LustreError: 134148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335814 with bad export cookie 11121898373030553779 [ 3919.990090] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.996800] LustreError: 134148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.345317] Lustre: server umount lustre-MDT0001 complete [ 3927.977549] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:30:19 (1763335819) [ 3933.020760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3948.576673] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3953.700369] LustreError: 136317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 3979.360325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.364924] Lustre: Skipped 2 previous similar messages [ 3984.777920] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 3990.547596] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3996.569711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.052558] LustreError: 136344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3997.219882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4000.640534] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4008.903517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4012.935505] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4018.154963] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4021.407939] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4024.593451] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4024.733675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4027.856968] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4028.037172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4029.925687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4029.927766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.941050] Lustre: Skipped 3 previous similar messages [ 4029.950866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.954557] Lustre: Skipped 7 previous similar messages [ 4035.753919] LustreError: 139094:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4035.762624] LustreError: 139094:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4035.839280] Lustre: server umount lustre-MDT0000 complete [ 4039.382249] LustreError: 136326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335933 with bad export cookie 11121898373030554801 [ 4039.389103] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.392831] LustreError: 136326:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4039.697692] Lustre: server umount lustre-MDT0001 complete [ 4053.806655] Lustre: server umount lustre-OST0000 complete [ 4062.315289] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:32:34 (1763335954) [ 4069.328543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4084.963851] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4090.080490] LustreError: 140144:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 4115.808570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.819248] Lustre: Skipped 2 previous similar messages [ 4121.760986] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4127.553342] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4134.334962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4134.749853] LustreError: 140170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.764513] LustreError: 140170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4134.937117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4138.805945] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4146.751285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4150.730951] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4156.224545] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4160.082868] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4167.664471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.783744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4167.914668] Lustre: lustre-OST0001: new disk, initializing [ 4167.917900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4173.876767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4173.889152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4174.047712] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4174.576035] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4182.516992] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4186.798324] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4187.073225] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4190.951771] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4191.189291] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4194.803396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.821542] Lustre: Skipped 2 previous similar messages [ 4194.827358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.830245] Lustre: Skipped 3 previous similar messages [ 4199.410564] LustreError: 143886:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4199.422172] LustreError: 143886:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4199.495139] Lustre: server umount lustre-OST0000 complete [ 4205.029518] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4205.045714] LustreError: Skipped 1 previous similar message [ 4209.431833] Lustre: server umount lustre-OST0001 complete [ 4219.484807] Lustre: server umount lustre-MDT0000 complete [ 4223.020778] LustreError: 140153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336117 with bad export cookie 11121898373030555746 [ 4223.024752] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.047384] LustreError: 140153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4223.382525] Lustre: server umount lustre-MDT0001 complete [ 4228.323463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4231.915031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4235.484109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4239.341178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4246.859701] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:35:38 (1763336138) [ 4248.835855] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4250.894211] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4253.332340] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:35:44 (1763336144) [ 4260.720405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.010323] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4261.031656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4266.128570] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4274.102122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4274.328275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4274.382806] Lustre: 146129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4278.051381] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4283.705396] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4287.321934] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4293.888124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4294.020409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4299.441180] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4301.466497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4305.041134] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4309.056954] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4309.203545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4312.362453] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4312.558405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4316.668789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4316.674300] LustreError: Skipped 1 previous similar message [ 4316.676538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4316.681573] Lustre: Skipped 5 previous similar messages [ 4316.686705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4316.689532] Lustre: Skipped 5 previous similar messages [ 4320.828983] Lustre: server umount lustre-OST0000 complete [ 4343.968798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4350.294254] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4355.722914] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4357.753144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.772732] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4360.526843] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4360.743616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.537475] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4367.019410] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.036059] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4367.926733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 4371.884535] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96551021e800.ost_server_uuid 50 [ 4373.614758] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96551021e800.ost_server_uuid in FULL state after 0 sec [ 4373.911454] LustreError: 147266:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.55@tcp) returned error from blocking AST (req@000000004bcfddb6 x1848991198457216 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000010d575ae/0x9a58eba4d87b39b5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.55@tcp remote: 0x56eddcda56cef7b0 expref: 5 pid: 147266 timeout: 4473 lvb_type: 0 [ 4373.928759] LustreError: lustre-MDT0000: A client on nid 192.168.204.55@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4373.937504] LustreError: 146124:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.55@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000010d575ae/0x9a58eba4d87b39b5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.55@tcp remote: 0x56eddcda56cef7b0 expref: 6 pid: 147266 timeout: 0 lvb_type: 0 [ 4383.584651] Lustre: server umount lustre-OST0000 complete [ 4387.300269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4387.303732] LustreError: Skipped 2 previous similar messages [ 4401.120267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4401.379969] Lustre: server umount lustre-MDT0000 complete [ 4403.177564] LustreError: 147267:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.219552] LustreError: 147267:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4405.862891] LustreError: 150908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336300 with bad export cookie 11121898373030556922 [ 4405.869595] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.882539] LustreError: 150908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4412.367240] Lustre: server umount lustre-MDT0001 complete [ 4423.204918] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4426.343213] Key type lgssc unregistered [ 4426.555519] LNet: 151643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4426.559429] LNetError: 151643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4426.583645] LNet: Removed LNI 192.168.204.155@tcp [ 4427.403654] Key type .llcrypt unregistered [ 4427.406876] Key type ._llcrypt unregistered [ 4450.032284] Key type ._llcrypt registered [ 4450.033807] Key type .llcrypt registered [ 4450.117878] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4451.035723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4451.057064] alg: No test for adler32 (adler32-zlib) [ 4452.124567] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4452.346601] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4453.968163] Key type lgssc registered [ 4454.888382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4461.757491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4461.778625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.297678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.808835] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4472.804328] LustreError: 153038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.817786] LustreError: 153038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4474.685835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4475.122864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.865766] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4484.634901] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4488.092161] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4494.566480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4494.900885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.108964] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4504.044662] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4508.220885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4513.586273] Lustre: Failing over lustre-MDT0000 [ 4513.768811] LustreError: 155285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4513.926828] Lustre: server umount lustre-MDT0000 complete [ 4514.842320] LustreError: 153032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.322737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4516.326282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.355408] LustreError: 153751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.370519] LustreError: 153751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.372627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.489042] LustreError: 153032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.507439] LustreError: 153032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4525.904846] LustreError: 153017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.55@tcp arrived at 1763336420 with bad export cookie 17574277123741446617 [ 4525.925971] LustreError: 153017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4532.651297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.837563] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4533.111146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.168498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4537.986199] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4538.347796] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4538.359378] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4540.226256] Lustre: lustre-MDT0000: Denying connection for new client a6bb9fcf-2500-4105-9218-d2c7bfbd0001 (at 192.168.204.55@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4548.580227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.590821] Lustre: Skipped 1 previous similar message [ 4548.593793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.597743] Lustre: Skipped 1 previous similar message [ 4553.711918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.714886] Lustre: Skipped 1 previous similar message [ 4554.507109] LustreError: 156222:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4554.519728] LustreError: 156222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4554.638067] Lustre: server umount lustre-OST0000 complete [ 4558.167381] LustreError: 156423:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4558.173056] Lustre: 155788:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4558.178352] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4558.191798] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4558.197116] LustreError: 155788:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000064a4f6a9 x1848992172387072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4558.217926] LustreError: 155788:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4558.226817] LustreError: 155788:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4558.232556] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 4558.376639] LustreError: 156423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4558.379610] LustreError: 156423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4558.474459] Lustre: server umount lustre-MDT0000 complete [ 4558.822947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4558.839135] Lustre: Skipped 1 previous similar message [ 4558.850088] LustreError: 154656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.871088] LustreError: 154656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4561.970560] LustreError: 153017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336456 with bad export cookie 17574277123741446974 [ 4561.972085] LustreError: 152346:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4561.983565] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4561.989154] LustreError: 153017:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4564.165888] LustreError: 156625:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4564.189628] LustreError: 156625:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4564.449925] Lustre: server umount lustre-MDT0001 complete [ 4575.648283] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4578.025240] Key type lgssc unregistered [ 4578.246093] LNet: 157157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4578.252655] LNetError: 157157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4578.273727] LNet: Removed LNI 192.168.204.155@tcp [ 4578.924618] Key type .llcrypt unregistered [ 4578.926048] Key type ._llcrypt unregistered [ 4600.055299] Key type ._llcrypt registered [ 4600.056804] Key type .llcrypt registered [ 4600.169447] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4600.858777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4601.000695] alg: No test for adler32 (adler32-zlib) [ 4601.988272] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4602.174164] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4603.832214] Key type lgssc registered [ 4604.776706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4610.824092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4610.856148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.304269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.864954] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4621.795707] LustreError: 158542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.813531] LustreError: 158542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4623.198979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4623.473030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4627.057366] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4632.490394] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4635.349604] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4640.702254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4640.960867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.897169] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4650.753951] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4657.338417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4662.753888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4662.761237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.773767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4662.785602] Lustre: Skipped 1 previous similar message [ 4662.805723] Lustre: Skipped 1 previous similar message [ 4662.972389] LustreError: 160789:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4663.140148] Lustre: server umount lustre-OST0000 complete [ 4667.875690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.894134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.898100] Lustre: Skipped 1 previous similar message [ 4672.719815] LustreError: 160989:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4672.736792] LustreError: 160989:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4672.895371] Lustre: server umount lustre-MDT0000 complete [ 4672.994310] LustreError: 159253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.329224] LustreError: 158522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336570 with bad export cookie 12581211808440991129 [ 4676.333235] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4676.337639] LustreError: 158522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4676.463701] LustreError: 161191:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4676.529517] LustreError: 161191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4676.821513] Lustre: server umount lustre-MDT0001 complete [ 4685.863581] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4688.685935] Key type lgssc unregistered [ 4689.036195] LNet: 161723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4689.048430] LNetError: 161723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4689.066195] LNet: Removed LNI 192.168.204.155@tcp [ 4689.670113] Key type .llcrypt unregistered [ 4689.671563] Key type ._llcrypt unregistered [ 4716.472460] Key type ._llcrypt registered [ 4716.474151] Key type .llcrypt registered [ 4716.564412] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4717.128979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4717.215949] alg: No test for adler32 (adler32-zlib) [ 4718.202027] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4718.336557] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4719.968157] Key type lgssc registered [ 4720.732467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.272444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4726.293949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.721475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.397948] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4736.535609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4736.827712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.176711] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4745.437804] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4748.876555] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4754.115872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4754.457609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.679471] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4765.222610] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4772.915823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4782.058618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4782.064425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4782.085951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.079153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4783.087237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4783.098958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4787.292224] LustreError: 165651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4787.431558] Lustre: server umount lustre-OST0000 complete [ 4793.315788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4793.318920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.329706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4793.340568] Lustre: Skipped 1 previous similar message [ 4793.353089] Lustre: Skipped 1 previous similar message [ 4796.786900] LustreError: 165853:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4796.793903] LustreError: 165853:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4796.931785] Lustre: server umount lustre-MDT0000 complete [ 4798.435769] LustreError: 164097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.442677] LustreError: 164097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4800.142793] LustreError: 163365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336694 with bad export cookie 1591021353336727612 [ 4800.150150] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4800.165065] LustreError: 163365:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4800.303342] LustreError: 166057:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4800.309520] LustreError: 166057:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4800.549528] Lustre: server umount lustre-MDT0001 complete [ 4809.761511] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4812.877364] Key type lgssc unregistered [ 4813.155878] LNet: 166590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4813.163458] LNetError: 166590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4814.188079] LNet: Removed LNI 192.168.204.155@tcp [ 4815.136761] Key type .llcrypt unregistered [ 4815.138962] Key type ._llcrypt unregistered [ 4834.477561] Key type ._llcrypt registered [ 4834.479966] Key type .llcrypt registered [ 4834.578866] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4835.821170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4835.839958] alg: No test for adler32 (adler32-zlib) [ 4836.900345] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4837.073509] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4838.744245] Key type lgssc registered [ 4839.955821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4851.195385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4851.253902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4852.759682] Lustre: *** cfs_fail_loc=135, val=0*** [ 4852.904199] LustreError: 168074:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4852.920995] LustreError: 168074:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.155@tcp: cfg command failed: rc = -2 [ 4852.931820] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4852.931820] [ 4852.956703] LustreError: MGC192.168.204.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4852.976558] LustreError: 168034:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4852.992038] LustreError: 168034:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4853.003843] LustreError: 168034:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4853.016238] LustreError: 168034:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4853.129943] Lustre: server umount lustre-MDT0000 complete [ 4853.134100] LustreError: 168034:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4859.614827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4860.116962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.955376] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4870.627004] LustreError: 168447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.642747] LustreError: 168447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4871.652978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4872.077270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4875.580328] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 4880.491319] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4884.011880] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4887.522750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.524435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4887.538399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.641654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4892.644488] Lustre: Skipped 1 previous similar message [ 4896.742743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.760498] Lustre: Skipped 1 previous similar message [ 4900.322779] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4900.422109] LustreError: 169789:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4900.444710] LustreError: 169789:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4900.565749] Lustre: server umount lustre-MDT0000 complete [ 4901.859741] LustreError: 169790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.883271] LustreError: 169790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4903.801058] LustreError: 168429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336798 with bad export cookie 9675257424780120913 [ 4903.822637] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4903.921319] LustreError: 169992:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4903.928725] LustreError: 169992:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4904.088815] Lustre: server umount lustre-MDT0001 complete [ 4913.202842] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4915.498290] Key type lgssc unregistered [ 4915.773685] LNet: 170525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4915.785244] LNetError: 170525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4915.804994] LNet: Removed LNI 192.168.204.155@tcp [ 4916.370815] Key type .llcrypt unregistered [ 4916.372400] Key type ._llcrypt unregistered [ 4933.766313] Key type ._llcrypt registered [ 4933.770336] Key type .llcrypt registered [ 4933.860175] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 4934.137114] Key type .llcrypt unregistered [ 4934.138811] Key type ._llcrypt unregistered [ 4948.779061] Key type ._llcrypt registered [ 4948.780614] Key type .llcrypt registered [ 4948.897505] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 4949.895819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4949.940216] alg: No test for adler32 (adler32-zlib) [ 4950.998910] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 4951.184369] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 4952.841867] Key type lgssc registered [ 4953.947681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4960.729270] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4960.747493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4976.352414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4981.472270] LustreError: 172616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.155@tcp: failed processing log, type 4: rc = -110 [ 5008.160208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5013.714187] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5018.613153] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5023.878424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.318522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5024.360039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5027.759653] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5029.357740] LustreError: 172642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.376031] LustreError: 172642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5034.468172] LustreError: 172642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.506520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5036.819676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5040.711071] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5045.871602] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5049.634799] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5052.277293] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5065.187871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5065.192273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.209231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.746204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5067.749319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.788456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5069.971142] LustreError: 175340:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5070.077784] Lustre: server umount lustre-OST0000 complete [ 5077.999459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5078.014068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.051874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.062238] Lustre: Skipped 2 previous similar messages [ 5080.857609] LustreError: 175543:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5080.861228] LustreError: 175543:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5080.940343] Lustre: server umount lustre-MDT0000 complete [ 5084.128903] LustreError: 173546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.155544] LustreError: 173546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5084.899856] LustreError: 172625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336979 with bad export cookie 4022760416338488788 [ 5084.904208] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5084.972494] LustreError: 172625:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5085.163729] LustreError: 175745:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5085.170527] LustreError: 175745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5085.468843] Lustre: server umount lustre-MDT0001 complete [ 5096.967519] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5100.422399] Key type lgssc unregistered [ 5100.776680] LNet: 176276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5100.787700] LNetError: 176276:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5101.862615] LNet: Removed LNI 192.168.204.155@tcp [ 5102.787192] Key type .llcrypt unregistered [ 5102.789755] Key type ._llcrypt unregistered [ 5125.809805] Key type ._llcrypt registered [ 5125.819825] Key type .llcrypt registered [ 5125.926558] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5127.058867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5127.089283] alg: No test for adler32 (adler32-zlib) [ 5128.091127] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5128.341863] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5129.992183] Key type lgssc registered [ 5131.157493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5137.982520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5138.032937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.618627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5143.648735] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5149.155742] LustreError: 177669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.182072] LustreError: 177669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5151.683974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5151.955440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5155.505481] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5160.056218] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5163.571821] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5169.852613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5170.225583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.786766] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5180.814539] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5184.628826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5190.453066] Lustre: Failing over lustre-MDT0000 [ 5190.618380] LustreError: 179968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5190.716802] Lustre: server umount lustre-MDT0000 complete [ 5192.776099] LustreError: 177665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.186949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5193.189632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.239584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.241703] LustreError: 177669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5195.327957] LustreError: 177669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5197.829194] LustreError: 177664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.943644] LustreError: 179291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.984306] LustreError: 179291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5208.824227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.933647] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5209.196866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5209.250690] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5213.188463] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5213.270644] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5214.210642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5214.232390] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5214.273623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5217.986516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5221.637123] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5229.552346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.556504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.562543] Lustre: Skipped 2 previous similar messages [ 5229.568330] Lustre: Skipped 1 previous similar message [ 5234.661690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5235.018104] LustreError: 181159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5235.030638] LustreError: 181159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5235.124698] Lustre: server umount lustre-OST0000 complete [ 5239.779043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5239.787247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.822378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.826330] Lustre: Skipped 2 previous similar messages [ 5244.898287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.903314] Lustre: Skipped 1 previous similar message [ 5250.020889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5250.029939] Lustre: Skipped 1 previous similar message [ 5253.088196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5253.185873] LustreError: 181360:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5253.190037] LustreError: 181360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5253.257222] Lustre: server umount lustre-MDT0000 complete [ 5255.141301] LustreError: 179291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5255.160169] LustreError: 179291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5256.293656] LustreError: 177651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337150 with bad export cookie 17607612067092275419 [ 5256.295257] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5256.305515] LustreError: 177651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5256.473733] LustreError: 181563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5256.476667] LustreError: 181563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5256.698545] Lustre: server umount lustre-MDT0001 complete [ 5266.358950] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5269.009425] Key type lgssc unregistered [ 5269.352915] LNet: 182094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5269.358368] LNetError: 182094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5269.375054] LNet: Removed LNI 192.168.204.155@tcp [ 5270.023803] Key type .llcrypt unregistered [ 5270.026160] Key type ._llcrypt unregistered [ 5291.109177] Key type ._llcrypt registered [ 5291.112128] Key type .llcrypt registered [ 5291.208604] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5291.885169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5291.904522] alg: No test for adler32 (adler32-zlib) [ 5292.965693] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5293.197159] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5294.861955] Key type lgssc registered [ 5295.985362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5303.040186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5303.064414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.506825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.980923] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5314.019766] LustreError: 183484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5314.035850] LustreError: 183484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5315.742979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5316.160802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.422449] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5324.839835] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5328.331797] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5334.395921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5334.728221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.881750] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5344.798855] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5349.108771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5351.649131] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5356.079892] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5364.957611] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5374.955067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.963250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.187247] LustreError: 185903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5378.367901] Lustre: server umount lustre-OST0000 complete [ 5382.625148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5382.628528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.655595] Lustre: Skipped 1 previous similar message [ 5382.666083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.669170] Lustre: Skipped 1 previous similar message [ 5385.190829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5385.194449] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.696109] LustreError: 186104:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5387.700381] LustreError: 186104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5387.775119] Lustre: server umount lustre-MDT0000 complete [ 5390.308326] LustreError: 184197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.336051] LustreError: 184197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5391.207725] LustreError: 183465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337285 with bad export cookie 7317800095253479594 [ 5391.208926] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5391.222215] LustreError: 183465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5391.414976] LustreError: 186305:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5391.422945] LustreError: 186305:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5391.667049] Lustre: server umount lustre-MDT0001 complete [ 5400.008798] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5402.844404] Key type lgssc unregistered [ 5403.109742] LNet: 186837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5403.119138] LNetError: 186837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5403.132147] LNet: Removed LNI 192.168.204.155@tcp [ 5403.919823] Key type .llcrypt unregistered [ 5403.926758] Key type ._llcrypt unregistered [ 5423.441524] Key type ._llcrypt registered [ 5423.443027] Key type .llcrypt registered [ 5423.585416] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5424.505516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5424.526425] alg: No test for adler32 (adler32-zlib) [ 5425.788953] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5426.077526] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5427.792325] Key type lgssc registered [ 5429.080512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5437.745075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5437.788714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5439.420708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.997342] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5448.689121] LustreError: 188211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.699775] LustreError: 188211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5451.917726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5452.309257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5456.153269] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5462.039116] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5466.122793] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.421187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5472.755108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.974844] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5483.239294] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5484.018189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5492.507936] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5501.957307] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5515.293753] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5520.236267] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5533.155221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5533.169794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.186825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.200958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.207527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5535.225563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5535.278902] Lustre: Skipped 1 previous similar message [ 5537.153099] LustreError: 191995:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5537.373495] Lustre: server umount lustre-OST0000 complete [ 5544.424182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5544.436807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.444031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5548.170815] LustreError: 192196:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5548.189507] LustreError: 192196:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5548.320155] Lustre: server umount lustre-MDT0000 complete [ 5550.565910] LustreError: 188211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.588075] LustreError: 188211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5552.125206] LustreError: 188191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337446 with bad export cookie 2985027965030987900 [ 5552.129791] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5552.141306] LustreError: 188191:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5552.282715] LustreError: 192399:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5552.287526] LustreError: 192399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5552.476321] Lustre: server umount lustre-MDT0001 complete [ 5561.324542] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5564.010578] Key type lgssc unregistered [ 5564.356850] LNet: 192930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5564.363889] LNetError: 192930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5564.380760] LNet: Removed LNI 192.168.204.155@tcp [ 5565.143351] Key type .llcrypt unregistered [ 5565.146219] Key type ._llcrypt unregistered [ 5585.588109] Key type ._llcrypt registered [ 5585.589864] Key type .llcrypt registered [ 5585.697799] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5586.391812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5586.486507] alg: No test for adler32 (adler32-zlib) [ 5587.514912] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5587.689940] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5589.344184] Key type lgssc registered [ 5590.313872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5597.100711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5597.126355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.582506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.243881] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5607.909396] LustreError: 194303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.922023] LustreError: 194303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5609.316318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5609.624059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.764953] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5618.051520] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5621.382415] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5627.683344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5628.015860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.549548] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5637.984596] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5643.386332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5659.109128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.126654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5659.136791] Lustre: Skipped 1 previous similar message [ 5659.143161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.381411] LustreError: 196700:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5661.486208] Lustre: server umount lustre-OST0000 complete [ 5665.761684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5665.772663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.799850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5665.809957] Lustre: Skipped 1 previous similar message [ 5669.347753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.359556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.878852] LustreError: 196902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5670.884771] LustreError: 196902:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5670.990618] Lustre: server umount lustre-MDT0000 complete [ 5674.141397] LustreError: 194285:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337568 with bad export cookie 15604021420899275790 [ 5674.151972] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5674.253845] LustreError: 197103:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5674.258615] LustreError: 197103:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5674.439231] Lustre: server umount lustre-MDT0001 complete [ 5682.756853] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5685.435319] Key type lgssc unregistered [ 5685.777686] LNet: 197636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5685.781788] LNetError: 197636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5685.801810] LNet: Removed LNI 192.168.204.155@tcp [ 5686.470569] Key type .llcrypt unregistered [ 5686.476336] Key type ._llcrypt unregistered [ 5703.843682] Key type ._llcrypt registered [ 5703.852299] Key type .llcrypt registered [ 5703.953518] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5704.819060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5704.897661] alg: No test for adler32 (adler32-zlib) [ 5705.896458] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5706.055271] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5707.712203] Key type lgssc registered [ 5708.885929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5716.616072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5716.642659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.261087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.233654] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5727.716174] LustreError: 198882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.727556] LustreError: 198882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5730.625676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5730.949742] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5734.705787] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5740.309565] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5744.544598] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5751.485598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5751.816334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5757.030854] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5762.380097] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5763.121794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5776.354499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.357476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.375497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5778.405125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.413372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5778.420356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.266423] LustreError: 201293:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5782.386182] Lustre: server umount lustre-OST0000 complete [ 5787.105665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5787.131261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.138840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.141815] Lustre: Skipped 1 previous similar message [ 5792.107866] LustreError: 201494:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5792.114852] LustreError: 201494:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5792.231273] Lustre: server umount lustre-MDT0000 complete [ 5793.768347] LustreError: 199920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5793.781559] LustreError: 199920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5796.095793] LustreError: 198864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337690 with bad export cookie 4988255138461566692 [ 5796.099135] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.122160] LustreError: 198864:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5796.326427] LustreError: 201696:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5796.332591] LustreError: 201696:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5796.505105] Lustre: server umount lustre-MDT0001 complete [ 5806.642736] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5809.422361] Key type lgssc unregistered [ 5809.732184] LNet: 202228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5809.741267] LNetError: 202228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5809.773265] LNet: Removed LNI 192.168.204.155@tcp [ 5810.529597] Key type .llcrypt unregistered [ 5810.531855] Key type ._llcrypt unregistered [ 5827.434045] Key type ._llcrypt registered [ 5827.436179] Key type .llcrypt registered [ 5827.550233] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 5828.538756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5828.557784] alg: No test for adler32 (adler32-zlib) [ 5829.642628] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 5829.918894] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 5831.625247] Key type lgssc registered [ 5832.908717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5839.831243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5839.849630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.234064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.330412] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5851.623177] LustreError: 203475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.644335] LustreError: 203475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5852.033864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5852.419444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.766461] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5860.677645] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5864.073447] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5870.129745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5870.466910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.636915] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5880.765468] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5886.849117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5895.373573] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5895.455445] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5895.754566] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5901.318156] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 5906.905572] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5908.763774] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5920.774509] Lustre: Permanently deactivating lustre-OST0001 [ 5920.792226] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5929.060150] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5938.113854] Lustre: Permanently reactivating lustre-OST0001 [ 5938.144943] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5938.156176] Lustre: Skipped 2 previous similar messages [ 5946.457774] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.465456] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5946.471862] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5946.477669] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5946.523441] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5951.422655] LustreError: 206948:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5951.469730] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.476022] Lustre: Skipped 1 previous similar message [ 5951.491677] LustreError: 205071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5951.543215] Lustre: server umount lustre-OST0001 complete [ 5956.581819] LustreError: 205992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.582763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.584114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5956.588834] LustreError: 205992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5956.596178] Lustre: Skipped 2 previous similar messages [ 5961.703113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.707416] LustreError: 205071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.775230] Lustre: Skipped 2 previous similar messages [ 5961.807092] LustreError: 205071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5966.829910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.830143] LustreError: 205071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.832544] Lustre: Skipped 1 previous similar message [ 5966.859832] LustreError: 205071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5969.376152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5969.408594] LustreError: 207149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5969.416024] LustreError: 207149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5969.538392] Lustre: server umount lustre-OST0000 complete [ 5975.521402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5975.531932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.545235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5979.232636] LustreError: 207352:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5979.234958] LustreError: 207352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5979.324049] Lustre: server umount lustre-MDT0000 complete [ 5982.182607] LustreError: 205794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.196938] LustreError: 205794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5982.732739] LustreError: 203456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337877 with bad export cookie 7468138104163693848 [ 5982.737263] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5982.746359] LustreError: 203456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5982.881040] LustreError: 207554:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5982.884691] LustreError: 207554:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5983.108389] Lustre: server umount lustre-MDT0001 complete [ 5993.171331] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 5995.895742] Key type lgssc unregistered [ 5996.140713] LNet: 208086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5996.148167] LNetError: 208086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5996.160555] LNet: Removed LNI 192.168.204.155@tcp [ 5996.777496] Key type .llcrypt unregistered [ 5996.779352] Key type ._llcrypt unregistered [ 6023.896532] Key type ._llcrypt registered [ 6023.898189] Key type .llcrypt registered [ 6024.026742] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 6036.681305] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6037.895180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6037.956913] alg: No test for adler32 (adler32-zlib) [ 6039.103247] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6039.368902] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6041.056178] Key type lgssc registered [ 6042.226840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6051.334917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6057.330615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6063.744043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6070.609716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.677759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6076.722047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6076.733890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.972184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6078.033141] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6078.080695] Lustre: lustre-MDT0000: new disk, initializing [ 6078.234876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6078.262491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6081.361334] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6091.026594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.091367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6091.153551] Lustre: 212009:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6091.205863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6091.209074] Lustre: Skipped 1 previous similar message [ 6091.255836] Lustre: lustre-MDT0001: new disk, initializing [ 6091.333421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6091.370208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6091.384361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6095.242113] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6101.927629] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6105.748487] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6112.628925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6112.727405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6113.050395] Lustre: lustre-OST0000: new disk, initializing [ 6113.054444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6113.147472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.830977] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6120.004454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6120.014418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6120.154959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6126.141424] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6129.974327] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6130.168176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6133.746103] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6133.971363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6140.907591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.911449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6140.911933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6140.919435] Lustre: Skipped 1 previous similar message [ 6140.925312] LustreError: Skipped 1 previous similar message [ 6146.032442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6146.044194] Lustre: Skipped 1 previous similar message [ 6150.114112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6150.170094] LustreError: 214949:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6150.273823] Lustre: server umount lustre-OST0000 complete [ 6156.261180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.285100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.287665] Lustre: Skipped 1 previous similar message [ 6159.994561] LustreError: 215152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6160.004807] LustreError: 215152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6160.094284] Lustre: server umount lustre-MDT0000 complete [ 6163.610844] LustreError: 212790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338058 with bad export cookie 6584129018869195579 [ 6163.615932] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.622833] LustreError: 212790:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6163.772657] LustreError: 215353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6163.782873] LustreError: 215353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6163.982705] Lustre: server umount lustre-MDT0001 complete [ 6172.001552] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 19:07:44 (1763338064) [ 6178.056837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.338267] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.346265] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6178.417627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.672774] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6183.916286] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.005825] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.188478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6191.672164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6195.715532] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6201.064776] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6204.253853] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6210.929236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6211.391398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.178649] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6220.405749] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6227.442848] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6230.656653] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6239.146845] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6245.436994] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6254.458636] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6275.695654] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6275.703188] Lustre: Skipped 2 previous similar messages [ 6308.952406] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6308.963500] Lustre: Skipped 3 previous similar messages [ 6382.130339] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6382.134200] Lustre: Skipped 8 previous similar messages [ 6395.873458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.879557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.884892] Lustre: Skipped 1 previous similar message [ 6395.887868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.901507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6401.275724] LustreError: 219472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6401.281904] LustreError: 219472:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6401.407137] Lustre: server umount lustre-OST0000 complete [ 6406.625146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6406.629183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.635018] Lustre: Skipped 1 previous similar message [ 6406.638170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6406.641040] Lustre: Skipped 2 previous similar messages [ 6411.062643] LustreError: 219678:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6411.081054] LustreError: 219678:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6411.154938] Lustre: server umount lustre-MDT0000 complete [ 6412.259366] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.284903] LustreError: 216024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6414.828909] LustreError: 218552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338309 with bad export cookie 6584129018869196461 [ 6414.832877] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.841054] LustreError: 218552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6415.143410] Lustre: server umount lustre-MDT0001 complete [ 6425.767451] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6428.192408] Key type lgssc unregistered [ 6428.489851] LNet: 220410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6428.498460] LNetError: 220410:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6429.542451] LNet: Removed LNI 192.168.204.155@tcp [ 6430.322841] Key type .llcrypt unregistered [ 6430.324746] Key type ._llcrypt unregistered [ 6451.234901] Key type ._llcrypt registered [ 6451.236792] Key type .llcrypt registered [ 6451.341963] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6452.530717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6452.561092] alg: No test for adler32 (adler32-zlib) [ 6453.664695] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6453.880629] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6455.560184] Key type lgssc registered [ 6456.746719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6464.543328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6464.567263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.108729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6469.644790] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6476.258391] LustreError: 221805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.271594] LustreError: 221805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6478.087672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6478.570584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.003595] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6487.716126] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6491.467406] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6498.323868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6498.617650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.985611] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_default_debug -1 all [ 6509.666417] Lustre: DEBUG MARKER: oleg455-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6528.481173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6528.488071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.508437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6529.476833] LustreError: 224211:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6529.637841] Lustre: server umount lustre-OST0000 complete [ 6534.628117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.630720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.634286] Lustre: Skipped 2 previous similar messages [ 6534.636241] Lustre: Skipped 1 previous similar message [ 6539.023410] LustreError: 224416:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6539.035571] LustreError: 224416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6539.141820] Lustre: server umount lustre-MDT0000 complete [ 6539.746324] LustreError: 221805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6539.767402] LustreError: 221805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6542.896243] LustreError: 221787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338437 with bad export cookie 6672954443564908296 [ 6542.903312] LustreError: MGC192.168.204.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.906982] LustreError: 221787:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6543.049457] LustreError: 224617:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6543.056181] LustreError: 224617:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6543.297521] Lustre: server umount lustre-MDT0001 complete [ 6555.028859] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6557.864677] Key type lgssc unregistered [ 6558.257685] LNet: 225152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.268136] LNetError: 225152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6558.290230] LNet: Removed LNI 192.168.204.155@tcp [ 6559.292817] Key type .llcrypt unregistered [ 6559.294540] Key type ._llcrypt unregistered [ 6578.150795] Key type ._llcrypt registered [ 6578.152339] Key type .llcrypt registered [ 6578.276642] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6578.530299] Key type .llcrypt unregistered [ 6578.532642] Key type ._llcrypt unregistered [ 6604.576863] Key type ._llcrypt registered [ 6604.581307] Key type .llcrypt registered [ 6604.767124] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing set_hostid [ 6621.818648] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing load_modules_local [ 6622.760689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6622.786040] alg: No test for adler32 (adler32-zlib) [ 6623.833499] Lustre: Lustre: Build Version: 2.16.61_53_g0076932 [ 6624.049131] LNet: Added LNI 192.168.204.155@tcp [8/256/0/180] [ 6625.705670] Key type lgssc registered [ 6626.912876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6634.640324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6639.503525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6645.261123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6650.781738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.706674] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6335 sec ========== 19:15:44 (1763338544) [ 6654.522694] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:15:46 (1763338546) === [ 6658.224872] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:15:50 (1763338550) === [ 6692.645175] Lustre: DEBUG MARKER: oleg455-server.virtnet: executing unload_modules_local [ 6695.582196] Key type lgssc unregistered [ 6695.862658] LNet: 231266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6695.876602] LNetError: 231266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6695.897942] LNet: Removed LNI 192.168.204.155@tcp [ 6696.678894] Key type .llcrypt unregistered [ 6696.680765] Key type ._llcrypt unregistered