[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 787230405 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002402] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.012025] Yama: becoming mindful. [ 0.014028] SELinux: Initializing. [ 0.016101] *** VALIDATE selinux *** [ 0.031342] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.038475] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.040105] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.042058] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.043157] *** VALIDATE tmpfs *** [ 0.046140] *** VALIDATE proc *** [ 0.048185] *** VALIDATE cgroup *** [ 0.049022] *** VALIDATE cgroup2 *** [ 0.050000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.050000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.050012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.051040] Spectre V2 : User space: Vulnerable [ 0.052013] Speculative Store Bypass: Vulnerable [ 0.056265] debug: unmapping init [mem 0xffffffffa8e59000-0xffffffffa8e60fff] [ 0.059991] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.061990] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.063037] ... version: 2 [ 0.064031] ... bit width: 48 [ 0.065009] ... generic registers: 4 [ 0.066029] ... value mask: 0000ffffffffffff [ 0.067039] ... max period: 00007fffffffffff [ 0.068018] ... fixed-purpose events: 3 [ 0.069031] ... event mask: 000000070000000f [ 0.071236] rcu: Hierarchical SRCU implementation. [ 0.075295] smp: Bringing up secondary CPUs ... [ 0.076702] x86: Booting SMP configuration: [ 0.077040] .... node #0, CPUs: #1 #2 #3 [ 0.089206] smp: Brought up 1 node, 4 CPUs [ 0.091018] smpboot: Max logical packages: 1 [ 0.092103] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147803] node 0 deferred pages initialised in 47ms [ 0.155014] devtmpfs: initialized [ 0.158365] x86/mm: Memory block size: 128MB [ 0.170171] gcov: version magic: 0x41383552 [ 0.183522] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.184100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.194386] pinctrl core: initialized pinctrl subsystem [ 0.198278] [ 0.200019] ************************************************************* [ 0.204023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208019] ** ** [ 0.214022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.218028] ** ** [ 0.222020] ** This means that this kernel is built to expose internal ** [ 0.225028] ** IOMMU data structures, which may compromise security on ** [ 0.226018] ** your system. ** [ 0.243022] ** ** [ 0.268021] ** If you see this message and you are not debugging the ** [ 0.281020] ** kernel, report this immediately to your vendor! ** [ 0.303033] ** ** [ 0.334029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.368042] ************************************************************* [ 0.389014] NET: Registered protocol family 16 [ 0.390000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.392082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.397125] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.405100] cpuidle: using governor menu [ 0.424327] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.439000] PCI: Using configuration type 1 for base access [ 0.442775] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.456318] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.461833] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.468801] cryptd: max_cpu_qlen set to 1000 [ 0.477198] ACPI: Added _OSI(Module Device) [ 0.480018] ACPI: Added _OSI(Processor Device) [ 0.482019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.485018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.492048] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.505511] ACPI: Interpreter enabled [ 0.507069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.509014] ACPI: Using IOAPIC for interrupt routing [ 0.512175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.516656] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.536283] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.540048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.545024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.551816] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.560945] acpiphp: Slot [2] registered [ 0.564203] acpiphp: Slot [5] registered [ 0.568445] acpiphp: Slot [6] registered [ 0.571213] acpiphp: Slot [7] registered [ 0.574197] acpiphp: Slot [8] registered [ 0.576283] acpiphp: Slot [9] registered [ 0.579208] acpiphp: Slot [10] registered [ 0.582210] acpiphp: Slot [3] registered [ 0.583000] acpiphp: Slot [4] registered [ 0.587201] acpiphp: Slot [11] registered [ 0.590137] acpiphp: Slot [12] registered [ 0.593140] acpiphp: Slot [13] registered [ 0.595502] acpiphp: Slot [14] registered [ 0.598151] acpiphp: Slot [15] registered [ 0.601177] acpiphp: Slot [16] registered [ 0.604157] acpiphp: Slot [17] registered [ 0.607142] acpiphp: Slot [18] registered [ 0.610158] acpiphp: Slot [19] registered [ 0.613150] acpiphp: Slot [20] registered [ 0.615142] acpiphp: Slot [21] registered [ 0.616152] acpiphp: Slot [22] registered [ 0.618146] acpiphp: Slot [23] registered [ 0.619000] acpiphp: Slot [24] registered [ 0.619000] acpiphp: Slot [25] registered [ 0.619000] acpiphp: Slot [26] registered [ 0.619000] acpiphp: Slot [27] registered [ 0.619000] acpiphp: Slot [28] registered [ 0.619000] acpiphp: Slot [29] registered [ 0.619000] acpiphp: Slot [30] registered [ 0.619000] acpiphp: Slot [31] registered [ 0.625077] PCI host bridge to bus 0000:00 [ 0.628027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.632033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.637027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.641030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.642000] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.642000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.644180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.653373] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.662517] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.663000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.674056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.682019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.685019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.692022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.698295] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.701872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.705039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.709730] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.716716] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.744021] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.754838] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.777199] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.789018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.813021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.831019] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.865627] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.877025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.887026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.911019] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.925000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.937019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.944017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.963018] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.975000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.992033] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.002020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.030018] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 1.052039] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.098023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.118025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.151030] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 1.175109] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.196019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.226021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.275026] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 1.305687] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.313431] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.320397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.324373] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.328256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.333129] iommu: Default domain type: Passthrough [ 1.344403] SCSI subsystem initialized [ 1.345202] ACPI: bus type USB registered [ 1.347098] usbcore: registered new interface driver usbfs [ 1.348073] usbcore: registered new interface driver hub [ 1.349085] usbcore: registered new device driver usb [ 1.350163] pps_core: LinuxPPS API ver. 1 registered [ 1.351010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.352065] PTP clock support registered [ 1.354393] EDAC MC: Ver: 3.0.0 [ 1.359179] PCI: Using ACPI for IRQ routing [ 1.363837] NetLabel: Initializing [ 1.366014] NetLabel: domain hash size = 128 [ 1.368012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.373093] NetLabel: unlabeled traffic allowed by default [ 1.386101] vgaarb: loaded [ 1.391024] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.394015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.406310] clocksource: Switched to clocksource kvm-clock [ 1.791509] VFS: Disk quotas dquot_6.6.0 [ 1.793358] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.795913] *** VALIDATE ramfs *** [ 1.797410] *** VALIDATE hugetlbfs *** [ 1.799055] pnp: PnP ACPI init [ 1.801497] pnp: PnP ACPI: found 6 devices [ 1.818466] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.825984] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.830772] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.836335] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.843586] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.849629] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.858112] NET: Registered protocol family 2 [ 1.873352] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.882398] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.887164] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.895148] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.898632] TCP: Hash tables configured (established 65536 bind 65536) [ 1.901707] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.910554] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.919456] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.930934] NET: Registered protocol family 1 [ 1.955111] RPC: Registered named UNIX socket transport module. [ 1.961642] RPC: Registered udp transport module. [ 1.966928] RPC: Registered tcp transport module. [ 1.974734] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.980815] NET: Registered protocol family 44 [ 1.985673] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.993147] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.002403] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.013551] pci 0000:00:01.0: quirk_isa_dma_hangs+0x0/0x20 took 10887 usecs [ 2.030201] PCI: CLS 0 bytes, default 64 [ 2.037864] Unpacking initramfs... [ 8.067802] debug: unmapping init [mem 0xffff8be8bcc54000-0xffff8be8bffbffff] [ 8.080238] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 8.086912] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 8.095581] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.697542] Initialise system trusted keyrings [ 9.704139] Key type blacklist registered [ 9.719408] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.760270] zbud: loaded [ 9.776931] *** VALIDATE nfs *** [ 9.781722] *** VALIDATE nfs4 *** [ 9.790229] pstore: using deflate compression [ 9.802269] Platform Keyring initialized [ 10.054607] NET: Registered protocol family 38 [ 10.056480] Key type asymmetric registered [ 10.058219] Asymmetric key parser 'x509' registered [ 10.060166] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 10.063649] io scheduler mq-deadline registered [ 10.065422] io scheduler kyber registered [ 10.066934] io scheduler bfq registered [ 10.100674] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 10.110748] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 10.113796] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 10.117336] ACPI: Power Button [PWRF] [ 10.381217] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 10.822873] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 11.736402] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 12.155850] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 12.920023] hrtimer: interrupt took 3840308 ns [ 13.118876] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 13.170948] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 13.264030] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 13.278287] Non-volatile memory driver v1.3 [ 13.284230] Linux agpgart interface v0.103 [ 13.683470] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 13.692857] vda: detected capacity change from 0 to 68526080 [ 13.741308] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 13.746443] vdb: detected capacity change from 0 to 1073741824 [ 13.811990] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.822199] vdc: detected capacity change from 0 to 2621440000 [ 13.894598] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 13.915682] vdd: detected capacity change from 0 to 2621440000 [ 14.246340] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.251029] vde: detected capacity change from 0 to 4294967296 [ 14.297495] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.304279] vdf: detected capacity change from 0 to 4294967296 [ 14.342996] libphy: Fixed MDIO Bus: probed [ 14.757846] usbcore: registered new interface driver usbserial_generic [ 14.863704] usbserial: USB Serial support registered for generic [ 14.887661] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 14.917446] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 14.931546] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 15.015619] mousedev: PS/2 mouse device common for all mice [ 15.024132] rtc_cmos 00:05: RTC can wake from S4 [ 15.032440] rtc_cmos 00:05: registered as rtc0 [ 15.036101] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 15.039172] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 15.039211] intel_pstate: CPU model not supported [ 15.046916] hid: raw HID events driver (C) Jiri Kosina [ 15.054693] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 15.059288] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 15.064786] usbcore: registered new interface driver usbhid [ 15.086245] usbhid: USB HID core driver [ 15.089207] drop_monitor: Initializing network drop monitor service [ 15.091477] Initializing XFRM netlink socket [ 15.094646] NET: Registered protocol family 10 [ 15.114240] Segment Routing with IPv6 [ 15.117388] NET: Registered protocol family 17 [ 15.124063] mpls_gso: MPLS GSO support [ 15.132345] RAS: Correctable Errors collector initialized. [ 15.137170] AVX version of gcm_enc/dec engaged. [ 15.138589] AES CTR mode by8 optimization enabled [ 16.475055] sched_clock: Marking stable (16475031879, 0)->(18740987386, -2265955507) [ 16.679153] registered taskstats version 1 [ 16.687527] Loading compiled-in X.509 certificates [ 16.694719] zswap: loaded using pool lzo/zbud [ 16.765527] Key type big_key registered [ 17.028394] Key type encrypted registered [ 17.033498] ima: No TPM chip found, activating TPM-bypass! [ 17.038181] ima: Allocated hash algorithm: sha1 [ 17.044499] ima: No architecture policies found [ 17.046530] evm: Initialising EVM extended attributes: [ 17.049608] evm: security.selinux [ 17.054241] evm: security.ima [ 17.056138] evm: security.capability [ 17.063315] evm: HMAC attrs: 0x1 [ 17.067344] rtc_cmos 00:05: setting system clock to 2025-11-17 17:46:44 UTC (1763401604) [ 17.077558] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 17.089376] debug: unmapping init [mem 0xffffffffa8b82000-0xffffffffa8e58fff] [ 17.103144] Write protecting the kernel read-only data: 28672k [ 17.118495] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 17.130290] debug: unmapping init [mem 0xffffffffa7b14000-0xffffffffa7bfffff] [ 17.345526] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.519352] systemd[1]: Detected virtualization kvm. [ 17.641703] systemd[1]: Detected architecture x86-64. [ 17.649229] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 17.734124] systemd[1]: No hostname configured. [ 17.746564] systemd[1]: Set hostname to . [ 17.751920] random: systemd: uninitialized urandom read (16 bytes read) [ 17.760102] systemd[1]: Initializing machine ID from random generator. [ 17.940738] random: ln: uninitialized urandom read (6 bytes read) [ 18.481350] random: systemd: uninitialized urandom read (16 bytes read) [ 18.490335] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 18.519678] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 18.612575] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ 18.784773] urandom_read: 5 callbacks suppressed [ 18.784785] random: systemd: uninitialized urandom read (16 bytes read) Starting Apply Kernel Variables... [ 18.841670] random: systemd: uninitialized urandom read (16 bytes read) Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 21.086592] device-mapper: uevent: version 1.0.3 [ 21.088611] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 23.272154] virtio_net virtio0 ens2: renamed from eth0 [ 23.373043] random: fast init done [ 24.553158] scsi host0: ata_piix [ 24.639865] scsi host1: ata_piix [ 24.641292] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 24.643434] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 28.599885] random: crng init done [ 32.015966] dracut-initqueue[598]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 34.045745] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 38.144403] printk: systemd: 26 output lines suppressed due to ratelimiting [ 38.987973] SELinux: Disabled at runtime. [ 39.079734] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 39.095643] systemd[1]: Detected virtualization kvm. [ 39.103643] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 41.545770] systemd[1]: initrd-switch-root.service: Succeeded. [ 41.571767] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 41.587314] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 41.598103] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 41.612323] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 41.639889] systemd[1]: Starting Journal Service... Starting Journal Service... [ 41.676296] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ 41.859268] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 43.965464] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 45.318276] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 45.381442] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 47.304484] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 47.558319] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 53.537588] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 55.071055] NFS: Registering the id_resolver key type [ 55.072806] Key type id_resolver registered [ 55.074298] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg158-server login: [ 112.289596] libcfs: loading out-of-tree module taints kernel. [ 112.307279] Key type ._llcrypt registered [ 112.308801] Key type .llcrypt registered [ 112.389668] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 126.202489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 127.303616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 127.312497] alg: No test for adler32 (adler32-zlib) [ 128.523498] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 129.236282] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 130.959163] Key type lgssc registered [ 132.342367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.465819] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.555501] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.313585] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.859541] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.905900] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.011111] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 189.987516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.043234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.054751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.254820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.303064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.366699] Lustre: lustre-MDT0000: new disk, initializing [ 191.464645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.482087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.637245] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.497705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.536088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 204.584198] Lustre: 6523:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 204.610918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 204.615322] Lustre: Skipped 1 previous similar message [ 204.671901] Lustre: lustre-MDT0001: new disk, initializing [ 204.736828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.766136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 204.779958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.336204] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.203961] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.294223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.414676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.810475] Lustre: lustre-OST0000: new disk, initializing [ 219.825096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.882726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.054760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.060921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.152516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.772580] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.372320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 237.487541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 237.643198] Lustre: lustre-OST0001: new disk, initializing [ 237.645993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.738652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.926268] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.309626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.326572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.429796] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.766274] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.464360] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 272.525441] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.134764] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing yml_node [ 281.169832] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 283.419672] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 285.839093] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 287.472230] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Nov 17 12:51:13 EST 2025 [ 304.885883] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 306.539439] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.625302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.626505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.637406] Lustre: Skipped 3 previous similar messages [ 326.642173] Lustre: Skipped 2 previous similar messages [ 331.747229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.764442] Lustre: Skipped 4 previous similar messages [ 336.866982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.870523] Lustre: Skipped 2 previous similar messages [ 337.375534] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.558059] LustreError: 12227:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 337.711808] Lustre: server umount lustre-MDT0000 complete [ 342.065247] LustreError: 6515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763401929 with bad export cookie 14228062679382746467 [ 342.068668] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.094989] LustreError: 6515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 347.104490] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.107516] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.108351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.141073] LustreError: 6530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 347.148799] Lustre: Skipped 1 previous similar message [ 348.310913] LustreError: 12430:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 348.314748] LustreError: 12430:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 348.486679] Lustre: server umount lustre-MDT0001 complete [ 354.101107] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 354.118178] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 354.314967] Lustre: server umount lustre-OST0000 complete [ 358.698859] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 358.762378] LustreError: 12834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 358.928128] Lustre: server umount lustre-OST0001 complete [ 366.036877] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 375.839764] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 387.174746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.716967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 402.724766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 409.857156] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 417.713551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 417.804902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.343308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.408965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.567645] Lustre: lustre-MDT0000: new disk, initializing [ 418.715706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.750673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.632604] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 433.554570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 433.657683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 433.759300] Lustre: 15022:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.849788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.858512] Lustre: Skipped 1 previous similar message [ 433.957455] Lustre: lustre-MDT0001: new disk, initializing [ 434.123239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.171813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.186453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.250745] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 446.581222] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.093954] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 456.992873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 457.063801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 457.363227] Lustre: lustre-OST0000: new disk, initializing [ 457.366277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.453747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 458.774885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 458.791231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 458.961853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 464.185898] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 472.412507] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 477.119426] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 477.364370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 482.003391] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 482.301831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 484.858529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 484.861636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.876566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.879408] Lustre: Skipped 1 previous similar message [ 490.559587] LustreError: 17961:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 490.580235] LustreError: 17961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 490.732724] Lustre: server umount lustre-OST0000 complete [ 495.072932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.099578] Lustre: Skipped 1 previous similar message [ 495.104896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.107607] Lustre: Skipped 3 previous similar messages [ 501.195635] LustreError: 18163:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 501.202720] LustreError: 18163:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 501.279732] Lustre: server umount lustre-MDT0000 complete [ 504.882542] LustreError: 15015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402092 with bad export cookie 14228062679382748000 [ 504.892363] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.895600] LustreError: 15015:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 505.182866] Lustre: server umount lustre-MDT0001 complete [ 508.835262] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:54:54 (1763402094) [ 515.692638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.070226] LustreError: 18890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.145537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.137566] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 521.189198] LustreError: 18889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.305198] LustreError: 18889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.320353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 533.733917] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 539.810869] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 543.669485] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 550.047850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 555.616536] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 561.177086] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 570.854441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.860289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.863737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.863749] LustreError: Skipped 1 previous similar message [ 570.866600] Lustre: Skipped 2 previous similar messages [ 570.883591] Lustre: Skipped 3 previous similar messages [ 581.087709] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 581.144409] LustreError: 21134:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 581.156867] LustreError: 21134:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 581.267754] Lustre: server umount lustre-OST0000 complete [ 586.208743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.211249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 586.242450] Lustre: Skipped 1 previous similar message [ 591.510729] Lustre: server umount lustre-MDT0000 complete [ 595.962379] LustreError: 21339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402183 with bad export cookie 14228062679382748875 [ 595.977000] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.366883] Lustre: server umount lustre-MDT0001 complete [ 608.323176] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 611.326723] Key type lgssc unregistered [ 611.626179] LNet: 22077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.644460] LNetError: 22077:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.688370] LNet: Removed LNI 192.168.201.158@tcp [ 612.600452] Key type .llcrypt unregistered [ 612.608326] Key type ._llcrypt unregistered [ 637.488976] Key type ._llcrypt registered [ 637.491427] Key type .llcrypt registered [ 637.612218] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 638.605186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.695764] alg: No test for adler32 (adler32-zlib) [ 639.737032] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 639.977665] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 641.631201] Key type lgssc registered [ 642.652413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.732114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.754498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.163620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.444859] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 661.475167] LustreError: 23477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.489615] LustreError: 23477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 662.291507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 662.735371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 666.609244] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 672.867859] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 677.378773] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 685.719129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 686.265179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.734437] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 696.525231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 696.887590] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 712.167828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.170241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 712.175857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 712.194470] Lustre: Skipped 1 previous similar message [ 713.412704] LustreError: 26167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 713.570625] Lustre: server umount lustre-OST0000 complete [ 719.332837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 719.344501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.360111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 719.363299] Lustre: Skipped 1 previous similar message [ 722.399822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 722.406346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.685745] LustreError: 26368:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 723.690948] LustreError: 26368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 723.783573] Lustre: server umount lustre-MDT0000 complete [ 727.496208] LustreError: 24196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402314 with bad export cookie 13943886186224875774 [ 727.504331] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 727.513889] LustreError: 24196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 727.629171] LustreError: 26570:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 727.636466] LustreError: 26570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 727.787637] Lustre: server umount lustre-MDT0001 complete [ 737.647174] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 740.176950] Key type lgssc unregistered [ 740.544668] LNet: 27102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.554516] LNetError: 27102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.604949] LNet: Removed LNI 192.168.201.158@tcp [ 742.568704] Key type .llcrypt unregistered [ 742.570505] Key type ._llcrypt unregistered [ 763.835399] Key type ._llcrypt registered [ 763.837409] Key type .llcrypt registered [ 763.947484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 765.126825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.196288] alg: No test for adler32 (adler32-zlib) [ 766.179784] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 766.367806] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 768.071562] Key type lgssc registered [ 769.446209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.841896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 777.868823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.295651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.623667] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 789.474804] LustreError: 28497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 789.494430] LustreError: 28497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 792.394614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 792.805415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 797.629739] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 803.533177] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 807.772542] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 822.306835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 822.562159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.606512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 828.102658] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 834.611128] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 847.849574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 847.853128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 847.877931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.376143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.382068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.385062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.410171] Lustre: Skipped 1 previous similar message [ 852.034298] LustreError: 31092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 852.138567] Lustre: server umount lustre-OST0000 complete [ 859.619158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 859.620754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.629245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.641147] Lustre: Skipped 1 previous similar message [ 859.660851] Lustre: Skipped 1 previous similar message [ 861.688120] LustreError: 31294:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 861.692323] LustreError: 31294:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 861.771380] Lustre: server umount lustre-MDT0000 complete [ 864.737936] LustreError: 31006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.752925] LustreError: 31006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 865.416271] LustreError: 28478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402452 with bad export cookie 2844642372546196031 [ 865.419830] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.423158] LustreError: 28478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 865.570051] LustreError: 31495:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 865.572270] LustreError: 31495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 865.810660] Lustre: server umount lustre-MDT0001 complete [ 876.086598] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 879.013362] Key type lgssc unregistered [ 879.252201] LNet: 32041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.261447] LNetError: 32041:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.300286] LNet: Removed LNI 192.168.201.158@tcp [ 881.208368] Key type .llcrypt unregistered [ 881.209841] Key type ._llcrypt unregistered [ 903.148137] Key type ._llcrypt registered [ 903.149746] Key type .llcrypt registered [ 903.305322] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 904.267102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.295190] alg: No test for adler32 (adler32-zlib) [ 905.328313] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 905.560112] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 907.295222] Key type lgssc registered [ 908.550389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.264901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.287246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.594967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.274160] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 926.190387] LustreError: 33434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.207024] LustreError: 33434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 928.232520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 928.639149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 932.264487] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 938.036325] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 941.608764] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 948.083371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.510443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.855801] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 958.957262] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 960.813687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 973.804874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 973.813281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 973.828664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 974.307312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 976.355050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.361627] Lustre: Skipped 2 previous similar messages [ 977.723112] LustreError: 35708:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 977.837420] Lustre: server umount lustre-OST0000 complete [ 981.477633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.483573] Lustre: Skipped 2 previous similar messages [ 981.483831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.593337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.601415] Lustre: Skipped 2 previous similar messages [ 987.240141] LustreError: 35909:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 987.247819] LustreError: 35909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 987.310949] Lustre: server umount lustre-MDT0000 complete [ 990.198134] LustreError: 35054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402577 with bad export cookie 17007083579673482376 [ 990.207749] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.209064] LustreError: 35054:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 990.338754] LustreError: 36110:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 990.346484] LustreError: 36110:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 990.522448] Lustre: server umount lustre-MDT0001 complete [ 999.071306] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1001.350650] Key type lgssc unregistered [ 1001.587949] LNet: 36644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.599139] LNetError: 36644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.613156] LNet: Removed LNI 192.168.201.158@tcp [ 1002.373562] Key type .llcrypt unregistered [ 1002.376723] Key type ._llcrypt unregistered [ 1022.032209] Key type ._llcrypt registered [ 1022.033849] Key type .llcrypt registered [ 1022.142771] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1023.010029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.032530] alg: No test for adler32 (adler32-zlib) [ 1024.039733] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1024.210341] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1025.855193] Key type lgssc registered [ 1026.838901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.047719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.083804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.576367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.169594] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1044.969132] LustreError: 38016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.995418] LustreError: 38016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1046.038489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1046.419369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.333246] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1053.804943] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1056.989720] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1062.312073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1062.628627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.504700] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1072.103135] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1080.077177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1088.165237] Lustre: lustre-OST0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1090.530036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1090.530511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.554584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.578747] Lustre: Skipped 1 previous similar message [ 1093.246809] Lustre: lustre-OST0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1093.264155] Lustre: Skipped 1 previous similar message [ 1095.655931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.658702] Lustre: Skipped 1 previous similar message [ 1100.770462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.784439] Lustre: Skipped 2 previous similar messages [ 1102.305350] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.331940] LustreError: 40285:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1102.442833] Lustre: server umount lustre-OST0000 complete [ 1121.249862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.251526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.264204] Lustre: Skipped 1 previous similar message [ 1126.013163] LustreError: 40660:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1126.017817] LustreError: 40660:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1126.108508] Lustre: server umount lustre-MDT0000 complete [ 1126.369130] LustreError: 39633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.379550] LustreError: 39633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1129.168821] LustreError: 37997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402716 with bad export cookie 2050411177773128963 [ 1129.176068] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.176510] LustreError: 37997:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.412390] LustreError: 40862:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1135.416099] LustreError: 40862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1135.536483] Lustre: server umount lustre-MDT0001 complete [ 1142.417829] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1144.243414] Key type lgssc unregistered [ 1144.506646] LNet: 41394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.512493] LNetError: 41394:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.533253] LNet: Removed LNI 192.168.201.158@tcp [ 1145.158342] Key type .llcrypt unregistered [ 1145.159880] Key type ._llcrypt unregistered [ 1163.765643] Key type ._llcrypt registered [ 1163.769575] Key type .llcrypt registered [ 1163.855449] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1164.702529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1164.711369] alg: No test for adler32 (adler32-zlib) [ 1165.705466] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1165.901307] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1167.570301] Key type lgssc registered [ 1168.563874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1174.563396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1174.582742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.896485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.609446] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1185.007669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1185.240417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1187.857184] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1191.675392] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.319407] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1199.629145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1199.858017] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.877864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1203.893099] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1208.518102] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1220.227701] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1220.578839] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1220.584764] Lustre: Skipped 2 previous similar messages [ 1223.415355] LustreError: 45010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1223.514677] Lustre: server umount lustre-MDT0000 complete [ 1225.326840] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.334683] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1226.748505] LustreError: 42746:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402814 with bad export cookie 5123851743281172734 [ 1226.757410] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.440750] LustreError: 42761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.455342] LustreError: 42761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1230.459975] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 1230.463607] Lustre: Skipped 3 previous similar messages [ 1230.819605] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.841309] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.848208] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1231.853376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1231.876761] Lustre: Skipped 1 previous similar message [ 1235.562273] LustreError: 42760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.572824] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 1240.676208] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.682223] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 1240.711155] LustreError: 44830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1240.728500] Lustre: Skipped 1 previous similar message [ 1241.056883] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1241.133953] LustreError: 45213:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1241.139796] LustreError: 45213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1241.288369] Lustre: server umount lustre-MDT0001 complete [ 1255.406847] LustreError: 45439:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1255.411994] LustreError: 45439:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1255.544270] Lustre: server umount lustre-OST0000 complete [ 1269.425826] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1271.544399] Key type lgssc unregistered [ 1271.764456] LNet: 46278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.773875] LNetError: 46278:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.784328] LNet: Removed LNI 192.168.201.158@tcp [ 1272.588256] Key type .llcrypt unregistered [ 1272.590600] Key type ._llcrypt unregistered [ 1294.615678] Key type ._llcrypt registered [ 1294.617773] Key type .llcrypt registered [ 1294.738585] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1295.885746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.901029] alg: No test for adler32 (adler32-zlib) [ 1296.870429] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1297.033538] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1298.679203] Key type lgssc registered [ 1299.587823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1306.315816] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1306.340541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1321.887451] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1327.008223] LustreError: 47635:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 1353.633167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.675902] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1365.400644] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1531.618377] LustreError: 48344:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1531.695844] Lustre: server umount lustre-OST0000 complete [ 1542.116846] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1544.452848] Key type lgssc unregistered [ 1544.694751] LNet: 49180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.701594] LNetError: 49180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.714454] LNet: Removed LNI 192.168.201.158@tcp [ 1545.347058] Key type .llcrypt unregistered [ 1545.350415] Key type ._llcrypt unregistered [ 1561.999867] Key type ._llcrypt registered [ 1562.001151] Key type .llcrypt registered [ 1562.086935] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1562.809572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1562.872104] alg: No test for adler32 (adler32-zlib) [ 1563.913732] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1564.102391] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1565.807277] Key type lgssc registered [ 1566.907795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1573.359554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1573.400170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1574.822074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.273250] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1583.119080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1583.345468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1586.290533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1590.497473] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1593.043667] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1598.482169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1598.854692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.873768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1603.110724] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1607.140248] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1609.573508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.576529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.650500] LustreError: 52798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1614.722483] Lustre: server umount lustre-OST0000 complete [ 1619.426046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.432808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.440800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.448821] Lustre: Skipped 1 previous similar message [ 1624.294235] LustreError: 53002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1624.298405] LustreError: 53002:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1624.404495] Lustre: server umount lustre-MDT0000 complete [ 1625.061281] LustreError: 50551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.073162] LustreError: 50551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1627.350514] LustreError: 52799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403214 with bad export cookie 12174713120711760229 [ 1627.352942] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.360297] LustreError: 52799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1627.487911] LustreError: 53203:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1627.494300] LustreError: 53203:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1627.623414] Lustre: server umount lustre-MDT0001 complete [ 1636.066674] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1638.376474] Key type lgssc unregistered [ 1638.618720] LNet: 53735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.624870] LNetError: 53735:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.639552] LNet: Removed LNI 192.168.201.158@tcp [ 1639.361680] Key type .llcrypt unregistered [ 1639.363857] Key type ._llcrypt unregistered [ 1655.017528] Key type ._llcrypt registered [ 1655.019463] Key type .llcrypt registered [ 1655.087246] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1655.577625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.632071] alg: No test for adler32 (adler32-zlib) [ 1656.586672] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1656.750660] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1658.383175] Key type lgssc registered [ 1659.072517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1664.749581] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1664.770479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1680.351570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1685.472846] LustreError: 55078:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 1712.161648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.871563] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1721.836561] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1726.287930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.632771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.702650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1729.162266] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1733.093394] LustreError: 56004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.116029] LustreError: 56004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1735.856327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.184244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.926467] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1743.150993] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1745.964526] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1751.521889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.522714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.530333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1751.535085] Lustre: Skipped 1 previous similar message [ 1751.540522] Lustre: Skipped 1 previous similar message [ 1753.587237] LustreError: 57358:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1753.661268] Lustre: server umount lustre-OST0000 complete [ 1777.120592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1777.124635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.133104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1777.140280] Lustre: Skipped 1 previous similar message [ 1777.151904] Lustre: Skipped 1 previous similar message [ 1778.477292] LustreError: 57734:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1778.480512] LustreError: 57734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1778.542579] Lustre: server umount lustre-MDT0000 complete [ 1781.365858] LustreError: 55085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403368 with bad export cookie 13251237644671968343 [ 1781.374194] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1781.382751] LustreError: 55085:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1787.567920] LustreError: 57936:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1787.637276] LustreError: 57936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1787.833289] Lustre: server umount lustre-MDT0001 complete [ 1796.929738] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1799.232792] Key type lgssc unregistered [ 1799.489990] LNet: 58469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.499519] LNetError: 58469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.512610] LNet: Removed LNI 192.168.201.158@tcp [ 1800.312317] Key type .llcrypt unregistered [ 1800.314046] Key type ._llcrypt unregistered [ 1817.726740] Key type ._llcrypt registered [ 1817.728509] Key type .llcrypt registered [ 1817.824323] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1818.649704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.672758] alg: No test for adler32 (adler32-zlib) [ 1819.673719] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1819.848074] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1821.511173] Key type lgssc registered [ 1822.535851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1828.862234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1828.880687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.223043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.556354] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1839.188415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1839.673614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.190884] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1848.478759] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1852.002283] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1859.229671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1859.866246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.207955] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1872.449598] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1875.269859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1890.793938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.797161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.800715] LustreError: Skipped 1 previous similar message [ 1890.802576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1890.815710] Lustre: Skipped 1 previous similar message [ 1893.759665] LustreError: 62112:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1893.846488] Lustre: server umount lustre-OST0000 complete [ 1901.027170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.027812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.029182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1901.029188] Lustre: Skipped 1 previous similar message [ 1901.048155] Lustre: Skipped 1 previous similar message [ 1903.292972] LustreError: 62313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1903.297327] LustreError: 62313:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1903.401656] Lustre: server umount lustre-MDT0000 complete [ 1906.145827] LustreError: 59842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.167722] LustreError: 59842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1906.828681] LustreError: 59821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403494 with bad export cookie 650375941528363770 [ 1906.833719] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1906.840290] LustreError: 59821:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1906.976668] LustreError: 62515:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1906.989588] LustreError: 62515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1907.152031] Lustre: server umount lustre-MDT0001 complete [ 1915.860397] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1918.128054] Key type lgssc unregistered [ 1918.410298] LNet: 63048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.416379] LNetError: 63048:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.433630] LNet: Removed LNI 192.168.201.158@tcp [ 1919.150343] Key type .llcrypt unregistered [ 1919.152784] Key type ._llcrypt unregistered [ 1947.744677] Key type ._llcrypt registered [ 1947.747706] Key type .llcrypt registered [ 1947.852752] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1948.475956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.503792] alg: No test for adler32 (adler32-zlib) [ 1949.535027] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1949.741202] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1951.399190] Key type lgssc registered [ 1952.356049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.207490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1959.230875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1960.630375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.896327] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1971.168811] LustreError: 64716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.194052] LustreError: 64716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1971.220147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1971.498796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.347486] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1979.026377] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.738514] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1988.992472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1989.302987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.570684] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1998.275165] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2005.672490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2014.188541] Lustre: Failing over lustre-MDT0000 [ 2014.334624] LustreError: 66981:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2014.552553] Lustre: server umount lustre-MDT0000 complete [ 2015.385063] LustreError: 64712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.227714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.262756] Lustre: Skipped 2 previous similar messages [ 2020.453794] LustreError: 66872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.472828] LustreError: 66872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2022.811726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.895303] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.112716] Lustre: *** cfs_fail_loc=135, val=0*** [ 2023.239537] LustreError: 67491:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2023.244069] LustreError: 67491:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -2 [ 2023.254920] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2023.254920] [ 2023.266215] LustreError: MGC192.168.201.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2023.276097] LustreError: 67479:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2023.287722] LustreError: 67479:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2023.298425] LustreError: 67479:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2023.323839] Lustre: server umount lustre-MDT0000 complete [ 2023.326186] LustreError: 67479:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2025.582389] LustreError: 66872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.613678] LustreError: 66872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2029.025382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.104613] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2029.116798] LustreError: 64708:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2029.365212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.415971] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2030.704556] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2032.301274] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2034.663841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2034.686303] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2034.733596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2037.728588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.733727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.744113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.777597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.780660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2039.801879] Lustre: Skipped 1 previous similar message [ 2043.775109] LustreError: 68299:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2043.777317] LustreError: 68299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2043.848880] Lustre: server umount lustre-OST0000 complete [ 2050.018075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2050.018789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2050.021604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2050.027708] Lustre: Skipped 1 previous similar message [ 2052.840443] LustreError: 68501:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2052.842995] LustreError: 68501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2052.902426] Lustre: server umount lustre-MDT0000 complete [ 2055.138392] LustreError: 66729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.149128] LustreError: 66729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2056.078865] LustreError: 64695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403643 with bad export cookie 10027903644019941146 [ 2056.084087] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.099040] LustreError: 64695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.231526] LustreError: 68702:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2056.235806] LustreError: 68702:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2056.370843] Lustre: server umount lustre-MDT0001 complete [ 2065.087787] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2067.534940] Key type lgssc unregistered [ 2067.750826] LNet: 69234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.757235] LNetError: 69234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.779920] LNet: Removed LNI 192.168.201.158@tcp [ 2069.316159] Key type .llcrypt unregistered [ 2069.318167] Key type ._llcrypt unregistered [ 2086.902299] Key type ._llcrypt registered [ 2086.903974] Key type .llcrypt registered [ 2086.964324] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2087.518603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.557738] alg: No test for adler32 (adler32-zlib) [ 2088.571210] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2088.740569] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2090.423818] Key type lgssc registered [ 2091.392755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2096.291605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2096.307796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.652351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.346161] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2106.953072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2107.275295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.129791] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2113.724978] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2116.016546] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2121.077599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2121.348781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.434693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2125.089851] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2129.302946] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2135.173536] Lustre: Failing over lustre-MDT0000 [ 2135.398144] LustreError: 72850:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2135.561240] Lustre: server umount lustre-MDT0000 complete [ 2137.060742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2137.064204] LustreError: 70602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.070622] Lustre: Skipped 2 previous similar messages [ 2137.092700] LustreError: 70602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2142.176704] LustreError: 70605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.187402] LustreError: 70605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2143.741834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.860896] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.031515] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2144.106663] LustreError: 73361:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2144.111475] LustreError: 73361:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -9 [ 2144.119601] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2144.119601] [ 2144.128228] LustreError: MGC192.168.201.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2144.136473] LustreError: 73349:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2144.140348] LustreError: 73349:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2144.146135] LustreError: 73349:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2144.161241] Lustre: server umount lustre-MDT0000 complete [ 2144.163230] LustreError: 73349:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2147.300807] LustreError: 70605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.320667] LustreError: 70605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2149.326385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.447379] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.646519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.687787] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2152.562177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2152.920655] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2154.989585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2155.011424] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2155.056401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2160.099254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.105412] Lustre: Skipped 1 previous similar message [ 2160.105839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2160.116972] Lustre: Skipped 1 previous similar message [ 2161.350213] LustreError: 74172:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2161.364549] LustreError: 74172:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2161.550838] Lustre: server umount lustre-OST0000 complete [ 2165.221726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.226160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.241996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.245840] Lustre: Skipped 1 previous similar message [ 2170.337912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.904911] LustreError: 74374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2170.915476] LustreError: 74374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2171.133240] Lustre: server umount lustre-MDT0000 complete [ 2174.524262] LustreError: 70585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403761 with bad export cookie 4221689953288919881 [ 2174.534456] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.543808] LustreError: 70585:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2174.689207] LustreError: 74576:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2174.707037] LustreError: 74576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2174.928647] Lustre: server umount lustre-MDT0001 complete [ 2185.026577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2187.750866] Key type lgssc unregistered [ 2188.037851] LNet: 75107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2188.047821] LNetError: 75107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.068733] LNet: Removed LNI 192.168.201.158@tcp [ 2188.849099] Key type .llcrypt unregistered [ 2188.853485] Key type ._llcrypt unregistered [ 2199.576462] Key type ._llcrypt registered [ 2199.578025] Key type .llcrypt registered [ 2199.683934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2199.883544] Key type .llcrypt unregistered [ 2199.885357] Key type ._llcrypt unregistered [ 2218.640882] Key type ._llcrypt registered [ 2218.642701] Key type .llcrypt registered [ 2218.738116] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2219.577915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.593436] alg: No test for adler32 (adler32-zlib) [ 2220.567631] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2220.736150] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2222.399197] Key type lgssc registered [ 2223.357915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.647340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.665954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.138245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.607666] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2241.317661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2241.504154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2241.583446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2244.244636] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2248.361583] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2250.736230] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2255.345566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2255.616959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.619907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2260.315635] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2265.377486] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2276.325414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2276.332781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2276.336296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.359388] Lustre: Skipped 1 previous similar message [ 2278.900388] LustreError: 79197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2278.975033] Lustre: server umount lustre-OST0000 complete [ 2282.464408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2282.479153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.491676] Lustre: Skipped 1 previous similar message [ 2282.495820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.567903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.569027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.595237] Lustre: Skipped 1 previous similar message [ 2291.680842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2291.691545] Lustre: Skipped 1 previous similar message [ 2296.287219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2296.350637] LustreError: 79399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2296.355810] LustreError: 79399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2296.411217] Lustre: server umount lustre-MDT0000 complete [ 2299.302742] LustreError: 76930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403886 with bad export cookie 15950887075294660189 [ 2299.304282] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2299.310656] LustreError: 76930:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2299.423595] LustreError: 79601:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2299.429489] LustreError: 79601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2299.549819] Lustre: server umount lustre-MDT0001 complete [ 2307.669735] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2309.766542] Key type lgssc unregistered [ 2309.962734] LNet: 80133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.972967] LNetError: 80133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.017729] LNet: Removed LNI 192.168.201.158@tcp [ 2311.668223] Key type .llcrypt unregistered [ 2311.672437] Key type ._llcrypt unregistered [ 2329.879709] Key type ._llcrypt registered [ 2329.881604] Key type .llcrypt registered [ 2329.963484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2330.529808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.551761] alg: No test for adler32 (adler32-zlib) [ 2331.561642] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2331.778682] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2333.439208] Key type lgssc registered [ 2334.786975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.484780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.498754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2345.048508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.992352] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2355.176185] LustreError: 81507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.198182] LustreError: 81507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2355.998608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2356.242687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2359.498166] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2364.018258] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2366.945635] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2372.747860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2373.082807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.607448] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2381.643512] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2390.520852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2396.642106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.645983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.656267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2398.010204] LustreError: 83775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2398.145495] Lustre: server umount lustre-OST0000 complete [ 2402.272206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2402.281782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.298627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.860545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.861138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.984923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.988171] Lustre: Skipped 2 previous similar messages [ 2410.998545] LustreError: 83976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2411.017049] LustreError: 83976:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2411.096164] Lustre: server umount lustre-MDT0000 complete [ 2414.924963] LustreError: 81489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404002 with bad export cookie 4265259165227244387 [ 2414.928893] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.951602] LustreError: 81489:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2415.176578] LustreError: 84178:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2415.181735] LustreError: 84178:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2415.313971] Lustre: server umount lustre-MDT0001 complete [ 2423.862656] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2426.281493] Key type lgssc unregistered [ 2426.574198] LNet: 84709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.585568] LNetError: 84709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.595889] LNet: Removed LNI 192.168.201.158@tcp [ 2427.373942] Key type .llcrypt unregistered [ 2427.381372] Key type ._llcrypt unregistered [ 2447.934252] Key type ._llcrypt registered [ 2447.935972] Key type .llcrypt registered [ 2448.031562] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2448.882136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.978138] alg: No test for adler32 (adler32-zlib) [ 2449.984957] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2450.209215] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2451.855895] Key type lgssc registered [ 2452.983496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.210270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.236678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.654323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.677406] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2470.864802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2471.139542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.071269] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2478.324490] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.309624] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2487.200524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2487.507810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.200926] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2492.912163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2496.293260] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2506.221389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.230547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2506.244550] Lustre: Skipped 1 previous similar message [ 2506.256922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.257869] Lustre: Skipped 1 previous similar message [ 2507.650846] LustreError: 88330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2507.793383] Lustre: server umount lustre-OST0000 complete [ 2511.347694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.349350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.365972] Lustre: Skipped 1 previous similar message [ 2511.380797] Lustre: Skipped 1 previous similar message [ 2516.451289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.455367] Lustre: Skipped 1 previous similar message [ 2517.306813] LustreError: 88533:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2517.315599] LustreError: 88533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2517.440172] Lustre: server umount lustre-MDT0000 complete [ 2521.086572] LustreError: 86064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404108 with bad export cookie 8614272983292605215 [ 2521.108556] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2521.268691] LustreError: 88735:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2521.271672] LustreError: 88735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2521.452259] Lustre: server umount lustre-MDT0001 complete [ 2529.936277] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2532.670509] Key type lgssc unregistered [ 2532.958849] LNet: 89267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.963495] LNetError: 89267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.980773] LNet: Removed LNI 192.168.201.158@tcp [ 2533.583162] Key type .llcrypt unregistered [ 2533.585459] Key type ._llcrypt unregistered [ 2555.149140] Key type ._llcrypt registered [ 2555.150551] Key type .llcrypt registered [ 2555.301314] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2556.501523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.529117] alg: No test for adler32 (adler32-zlib) [ 2557.524437] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2557.702339] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2559.351182] Key type lgssc registered [ 2560.254825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2566.526300] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2566.539277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2582.111334] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2587.231258] LustreError: 90632:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 2613.919150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.544712] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2623.278804] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2627.991974] LustreError: 91489:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2628.079593] Lustre: server umount lustre-OST0000 complete [ 2636.515625] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:30:22 (1763404222) [ 2643.000415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.513792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.999883] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2648.549084] LustreError: 92161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.568118] LustreError: 92161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2652.646431] LustreError: 92161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.250636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2655.567897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2659.228473] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2664.297690] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2667.791412] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2674.361291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2674.737268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.579679] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2685.178398] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2688.063742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2700.257734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2700.265067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.281832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2703.328466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2703.332296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2703.398735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2706.122924] LustreError: 94565:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2706.125983] LustreError: 94565:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2706.302330] Lustre: server umount lustre-OST0000 complete [ 2712.032597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2712.045554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.058410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.066107] Lustre: Skipped 1 previous similar message [ 2718.689615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.700368] Lustre: Skipped 3 previous similar messages [ 2723.811057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2723.818192] Lustre: Skipped 1 previous similar message [ 2724.319235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2724.383680] LustreError: 94766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2724.396465] LustreError: 94766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2726.435080] LustreError: 94766:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2726.441514] LustreError: 94766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2726.484334] Lustre: server umount lustre-MDT0000 complete [ 2727.907263] LustreError: 92155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.929821] LustreError: 92155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2729.553326] LustreError: 92142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404316 with bad export cookie 7685918259659395060 [ 2729.554976] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2729.568657] LustreError: 92142:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2729.867744] Lustre: server umount lustre-MDT0001 complete [ 2739.141732] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2741.645977] Key type lgssc unregistered [ 2741.942536] LNet: 95503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2741.952362] LNetError: 95503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2741.976492] LNet: Removed LNI 192.168.201.158@tcp [ 2742.808324] Key type .llcrypt unregistered [ 2742.813439] Key type ._llcrypt unregistered [ 2764.159678] Key type ._llcrypt registered [ 2764.161251] Key type .llcrypt registered [ 2764.244584] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2765.191900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2765.240271] alg: No test for adler32 (adler32-zlib) [ 2766.304446] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2766.525405] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2768.231161] Key type lgssc registered [ 2769.399784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2777.579810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2777.622271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2779.115518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.061743] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2784.234321] LustreError: 96896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.259852] LustreError: 96896:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2789.346947] LustreError: 96896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.673711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2794.148581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2798.512540] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2804.271558] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2808.227563] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2814.549380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2814.823693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.331073] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2824.047809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2825.801422] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2834.402803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.410166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.416332] LustreError: Skipped 1 previous similar message [ 2834.429278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.521707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.535733] Lustre: Skipped 2 previous similar messages [ 2839.759872] LustreError: 99142:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2839.898848] Lustre: server umount lustre-OST0000 complete [ 2844.646424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.653252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.660823] Lustre: Skipped 2 previous similar messages [ 2844.697219] Lustre: Skipped 1 previous similar message [ 2849.761201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2849.765236] Lustre: Skipped 1 previous similar message [ 2849.835759] LustreError: 99344:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2849.838618] LustreError: 99344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2849.963849] Lustre: server umount lustre-MDT0000 complete [ 2853.675558] LustreError: 96876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404441 with bad export cookie 6368294350079144552 [ 2853.676398] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.690931] LustreError: 96876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2853.839546] LustreError: 99546:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2853.843977] LustreError: 99546:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2854.033200] Lustre: server umount lustre-MDT0001 complete [ 2863.508355] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2866.290212] Key type lgssc unregistered [ 2866.588370] LNet: 100079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.604253] LNetError: 100079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2867.623318] LNet: Removed LNI 192.168.201.158@tcp [ 2868.441175] Key type .llcrypt unregistered [ 2868.443123] Key type ._llcrypt unregistered [ 2885.155122] Key type ._llcrypt registered [ 2885.156796] Key type .llcrypt registered [ 2885.269484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2886.292948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2886.314887] alg: No test for adler32 (adler32-zlib) [ 2887.334656] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2887.518515] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2889.167317] Key type lgssc registered [ 2890.422328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2898.333770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2898.359556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2913.954526] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2919.135502] LustreError: 101302:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 2945.888684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.860975] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2956.392967] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.396610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.671297] LustreError: MGC192.168.201.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2962.683054] LustreError: 102214:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2962.691098] LustreError: 102214:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2962.700709] LustreError: 102214:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2962.706745] LustreError: 101311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404550 with bad export cookie 7182171028593069788 [ 2962.720943] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2962.785720] Lustre: server umount lustre-MDT0000 complete [ 2962.787810] LustreError: 102214:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2970.024714] LustreError: 102646:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2970.124094] Lustre: server umount lustre-OST0000 complete [ 2978.018443] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 2986.579346] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2995.200891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3001.933227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.708568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.126982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.051330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3024.141551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.468304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3024.500548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3024.571695] Lustre: lustre-MDT0000: new disk, initializing [ 3024.681412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.720877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3028.453453] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3038.031141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3038.079408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3038.161210] Lustre: 104983:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3038.228292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3038.232810] Lustre: Skipped 1 previous similar message [ 3038.341554] Lustre: lustre-MDT0001: new disk, initializing [ 3038.467569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3038.506583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3038.518194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3042.321286] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3049.700904] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3053.866622] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3060.830427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3060.910442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3061.196983] Lustre: lustre-OST0000: new disk, initializing [ 3061.200441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3061.323331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.103573] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3070.588784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3070.593866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3070.789116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3074.887831] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3078.942695] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3079.125713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3082.684969] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3082.872478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3086.313885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3086.319673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.327431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3086.333097] Lustre: Skipped 1 previous similar message [ 3091.425093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3091.445281] Lustre: Skipped 1 previous similar message [ 3096.546983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.556097] Lustre: Skipped 1 previous similar message [ 3099.106143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3099.156470] LustreError: 107932:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3099.159367] LustreError: 107932:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3099.340622] Lustre: server umount lustre-OST0000 complete [ 3105.250932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3105.256143] LustreError: Skipped 1 previous similar message [ 3105.258416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.263806] Lustre: Skipped 1 previous similar message [ 3105.266530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.790912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.073670] LustreError: 108134:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3110.088123] LustreError: 108134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3110.290259] Lustre: server umount lustre-MDT0000 complete [ 3114.777566] LustreError: 106738:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404702 with bad export cookie 7182171028593069879 [ 3114.781912] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.816269] LustreError: 106738:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3114.924389] LustreError: 108336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3114.943841] LustreError: 108336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3115.218198] Lustre: server umount lustre-MDT0001 complete [ 3125.528957] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:38:30 (1763404710) [ 3129.529598] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3144.756412] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3153.234574] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3162.479204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3168.998597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3174.833309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3181.167166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3188.971114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.129154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.350522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3189.401527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3189.486623] Lustre: lustre-MDT0000: new disk, initializing [ 3189.598683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.630995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3194.136338] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3205.820342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3205.904590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3206.000388] Lustre: 111474:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3206.115582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3206.139545] Lustre: Skipped 1 previous similar message [ 3206.258761] Lustre: lustre-MDT0001: new disk, initializing [ 3206.374538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3206.434398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3206.443116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3211.138858] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3220.402622] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.355074] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.534510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.630639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3232.030745] Lustre: lustre-OST0000: new disk, initializing [ 3232.033832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3232.185057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.629792] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3241.570744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3241.581042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3241.706886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3248.802058] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.967299] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3255.435240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3261.297780] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3261.518776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3267.552194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.553900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.572527] LustreError: Skipped 1 previous similar message [ 3267.631730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.654091] Lustre: Skipped 3 previous similar messages [ 3277.803120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3277.807913] Lustre: Skipped 3 previous similar messages [ 3278.303148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3278.344347] LustreError: 114412:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3278.353383] LustreError: 114412:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3278.439218] Lustre: server umount lustre-OST0000 complete [ 3283.428523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3283.441390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.464420] Lustre: Skipped 1 previous similar message [ 3289.260038] LustreError: 114616:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3289.384282] LustreError: 114616:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3289.787845] Lustre: server umount lustre-MDT0000 complete [ 3293.152814] LustreError: 113227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.180080] LustreError: 113227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3293.726743] LustreError: 111465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404881 with bad export cookie 7182171028593070761 [ 3293.732037] LustreError: 111465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3293.741412] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.105699] Lustre: server umount lustre-MDT0001 complete [ 3303.867567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3304.450566] LustreError: 115300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3304.586788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.164981] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3310.050901] LustreError: 115300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.156645] LustreError: 115300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.170757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3322.304520] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3327.593467] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3330.981477] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3337.962182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3343.731903] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3349.059365] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3355.330927] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3359.716374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3359.716575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.724123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3359.727833] Lustre: Skipped 2 previous similar messages [ 3359.738664] Lustre: Skipped 4 previous similar messages [ 3361.023805] LustreError: 117643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3361.029169] LustreError: 117643:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3361.198881] Lustre: server umount lustre-OST0000 complete [ 3370.917159] Lustre: server umount lustre-MDT0000 complete [ 3374.669468] LustreError: 115281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404962 with bad export cookie 7182171028593071643 [ 3374.673378] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3374.684039] LustreError: 115281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3375.042420] Lustre: server umount lustre-MDT0001 complete [ 3384.787868] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3387.196981] Key type lgssc unregistered [ 3387.461142] LNet: 118579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3387.470346] LNetError: 118579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3388.518304] LNet: Removed LNI 192.168.201.158@tcp [ 3389.307956] Key type .llcrypt unregistered [ 3389.309548] Key type ._llcrypt unregistered [ 3405.809775] Key type ._llcrypt registered [ 3405.811969] Key type .llcrypt registered [ 3405.905525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 3419.708585] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3420.949354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3420.976798] alg: No test for adler32 (adler32-zlib) [ 3421.984506] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3422.248136] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3423.906162] Key type lgssc registered [ 3424.813930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3433.129526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3439.050576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3445.658584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3451.644927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.313510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.395185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3457.412555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.671904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3458.716313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3458.786878] Lustre: lustre-MDT0000: new disk, initializing [ 3458.879284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.900165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3462.003470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3472.253249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3472.311763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3472.412237] Lustre: 121702:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3472.462349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3472.469131] Lustre: Skipped 1 previous similar message [ 3472.541250] Lustre: lustre-MDT0001: new disk, initializing [ 3472.628634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.667572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3472.672348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3476.212813] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3483.866715] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3488.011535] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3495.686162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.822070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3496.120108] Lustre: lustre-OST0000: new disk, initializing [ 3496.131818] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3496.223260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.173339] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3506.277668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3506.283208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3506.444336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3509.426429] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.619461] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3513.905722] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3517.999533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3518.166379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3522.016077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3522.018353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.026260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3522.040228] Lustre: Skipped 1 previous similar message [ 3526.270145] LustreError: 124645:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3526.452674] Lustre: server umount lustre-OST0000 complete [ 3532.256670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.260357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.271400] Lustre: Skipped 1 previous similar message [ 3532.276944] Lustre: Skipped 1 previous similar message [ 3536.598579] LustreError: 124848:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3536.605212] LustreError: 124848:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3536.800862] Lustre: server umount lustre-MDT0000 complete [ 3540.779795] LustreError: 121696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405128 with bad export cookie 13476868200106436253 [ 3540.780421] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.797093] LustreError: 121696:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3540.977552] LustreError: 125049:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3540.980786] LustreError: 125049:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3541.170205] Lustre: server umount lustre-MDT0001 complete [ 3550.608261] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:45:36 (1763405136) [ 3557.712797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.073173] LustreError: 125723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.089624] LustreError: 125723:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3558.173101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.983710] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3563.487851] LustreError: 125723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.587418] LustreError: 125723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.928725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3570.381109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3574.310073] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3579.774685] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3583.094967] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3586.017069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3586.023578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.043532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.066433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.070511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.184460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.200364] Lustre: Skipped 2 previous similar messages [ 3599.331883] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3599.453236] LustreError: 127064:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3599.468598] LustreError: 127064:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3599.557837] Lustre: server umount lustre-MDT0000 complete [ 3600.353498] LustreError: 125722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3600.380279] LustreError: 125722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3603.204826] LustreError: 125703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405190 with bad export cookie 13476868200106437135 [ 3603.208589] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3603.210263] LustreError: 125703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3603.596739] Lustre: server umount lustre-MDT0001 complete [ 3611.719106] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:46:37 (1763405197) [ 3617.776705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.311932] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3638.431895] LustreError: 127912:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 3664.225574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.238989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3674.776668] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3676.742129] LustreError: 128578:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3676.744520] LustreError: 128578:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3676.877609] Lustre: server umount lustre-OST0000 complete [ 3684.001249] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:47:49 (1763405269) [ 3689.844417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.205342] LustreError: 129249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3690.260829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.555434] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3699.748300] LustreError: 129249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.771379] LustreError: 129249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3701.311342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3705.372398] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3710.320755] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3713.455249] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3719.298323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3724.265793] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3729.447851] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3742.693806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3742.704810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.722065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3742.727467] Lustre: Skipped 2 previous similar messages [ 3746.776815] LustreError: 131494:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3746.791813] LustreError: 131494:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3746.927665] Lustre: server umount lustre-MDT0000 complete [ 3750.196117] LustreError: 129228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405337 with bad export cookie 13476868200106437800 [ 3750.208791] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3750.372924] LustreError: 129245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.392719] LustreError: 129245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3750.523189] Lustre: server umount lustre-MDT0001 complete [ 3756.331364] Lustre: server umount lustre-OST0000 complete [ 3764.236726] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:49:10 (1763405350) [ 3770.096896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3770.470659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3770.476739] Lustre: Skipped 2 previous similar messages [ 3773.727720] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3781.967072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3786.460502] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3791.677682] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3795.194894] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3801.080982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3806.028994] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3810.986612] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3815.279831] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3815.802459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3816.754525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3820.478832] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3820.694181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3823.073050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.081027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.119291] Lustre: Skipped 3 previous similar messages [ 3823.130168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3823.142279] Lustre: Skipped 3 previous similar messages [ 3828.484218] LustreError: 135320:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3828.499795] LustreError: 135320:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3828.623715] Lustre: server umount lustre-OST0000 complete [ 3833.312728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3833.333718] LustreError: Skipped 1 previous similar message [ 3838.688660] Lustre: server umount lustre-MDT0000 complete [ 3841.506206] LustreError: 132568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3841.518115] LustreError: 132568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3841.971402] LustreError: 132548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405429 with bad export cookie 13476868200106438871 [ 3841.975935] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3841.978497] LustreError: 132548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3842.296257] Lustre: server umount lustre-MDT0001 complete [ 3850.547905] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:50:36 (1763405436) [ 3856.963546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3872.479589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3877.602943] LustreError: 136369:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 3903.327310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.331782] Lustre: Skipped 2 previous similar messages [ 3909.157410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3915.005399] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3921.683231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.940474] LustreError: 136396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.058663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3926.257665] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3935.222995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3939.402753] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3944.725168] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3949.392907] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3953.513520] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3953.746343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3957.906839] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3958.128586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3961.317482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3961.330160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3961.355523] Lustre: Skipped 3 previous similar messages [ 3961.372743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3961.375688] Lustre: Skipped 5 previous similar messages [ 3966.369638] LustreError: 139144:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3966.376053] LustreError: 139144:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3966.543965] Lustre: server umount lustre-MDT0000 complete [ 3969.989178] LustreError: 136378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405557 with bad export cookie 13476868200106439893 [ 3969.999898] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.391373] Lustre: server umount lustre-MDT0001 complete [ 3984.365306] Lustre: server umount lustre-OST0000 complete [ 3992.698812] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:52:58 (1763405578) [ 3999.522179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4015.219814] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4020.383549] LustreError: 140195:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 4046.111384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.114019] Lustre: Skipped 2 previous similar messages [ 4051.374975] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4056.385264] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.415855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.778694] LustreError: 140222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.793544] LustreError: 140222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4062.956664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4065.620486] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4072.088414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4075.456343] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4079.487258] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4082.889742] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4089.143497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4089.215406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4089.312761] Lustre: lustre-OST0001: new disk, initializing [ 4089.318466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4094.007398] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4099.630045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4099.638512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4099.688840] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4101.193838] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4104.643239] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4104.816618] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4107.709473] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4107.897914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4109.807980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.816634] Lustre: Skipped 2 previous similar messages [ 4109.821031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4109.823783] Lustre: Skipped 3 previous similar messages [ 4114.400342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.639105] LustreError: 143926:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4115.649719] LustreError: 143926:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4115.682721] Lustre: server umount lustre-OST0000 complete [ 4133.343155] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4133.487442] Lustre: server umount lustre-OST0001 complete [ 4151.263134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4151.470428] Lustre: server umount lustre-MDT0000 complete [ 4155.013236] LustreError: 140203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405742 with bad export cookie 13476868200106440838 [ 4155.019194] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.026418] LustreError: 140203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4155.325641] Lustre: server umount lustre-MDT0001 complete [ 4160.442132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4163.854153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.359973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.302067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.530166] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:56:02 (1763405762) [ 4178.292180] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4179.849490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4181.505662] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:56:07 (1763405767) [ 4188.013057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.311674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4188.340759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4191.934358] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4199.609412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4199.704867] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4199.741176] Lustre: 146174:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4203.994755] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4209.813777] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4213.360197] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4220.170591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4220.367154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4225.465813] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4226.638424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4229.568649] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4233.409678] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4233.596353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4237.771670] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4237.995167] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4241.891277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.894445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.902263] LustreError: Skipped 4 previous similar messages [ 4241.915267] Lustre: Skipped 6 previous similar messages [ 4241.917339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.922601] Lustre: Skipped 16 previous similar messages [ 4245.696056] LustreError: 148957:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4245.701874] LustreError: 148957:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4245.801601] Lustre: server umount lustre-OST0000 complete [ 4267.184438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4273.612282] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4281.087460] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4282.031772] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.055769] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4282.061253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.656606] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4285.869624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4290.243507] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4290.531319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4294.996751] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8fcd5842e800.ost_server_uuid 50 [ 4296.955828] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8fcd5842e800.ost_server_uuid in FULL state after 0 sec [ 4297.420582] LustreError: 146183:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.58@tcp) returned error from blocking AST (req@00000000ade794ab x1849064170957056 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000854fbefc/0xbb077439781a41d9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.58@tcp remote: 0x4357a1b1f9adbfc9 expref: 5 pid: 146181 timeout: 4397 lvb_type: 0 [ 4297.453655] LustreError: lustre-MDT0000: A client on nid 192.168.201.58@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4297.467896] LustreError: 146171:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.58@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000854fbefc/0xbb077439781a41d9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.58@tcp remote: 0x4357a1b1f9adbfc9 expref: 6 pid: 146181 timeout: 0 lvb_type: 0 [ 4308.567051] Lustre: server umount lustre-OST0000 complete [ 4312.544926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4312.552251] LustreError: Skipped 2 previous similar messages [ 4318.664298] Lustre: server umount lustre-MDT0000 complete [ 4322.351981] LustreError: 146166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405909 with bad export cookie 13476868200106442014 [ 4322.354493] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4322.367034] LustreError: 146166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4328.671421] Lustre: server umount lustre-MDT0001 complete [ 4340.047399] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4343.877568] Key type lgssc unregistered [ 4344.151629] LNet: 151668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4344.166208] LNetError: 151668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4344.190789] LNet: Removed LNI 192.168.201.158@tcp [ 4345.132599] Key type .llcrypt unregistered [ 4345.138326] Key type ._llcrypt unregistered [ 4370.877580] Key type ._llcrypt registered [ 4370.884951] Key type .llcrypt registered [ 4370.978084] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4371.870098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4371.945392] alg: No test for adler32 (adler32-zlib) [ 4373.113942] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4373.328882] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4375.111187] Key type lgssc registered [ 4376.435502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4383.932848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4383.963848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4385.354654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.931410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4395.499496] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.509341] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4399.419392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4399.813593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4404.573105] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4410.648903] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4414.808401] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4422.618155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4422.977389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.748406] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4430.172829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4435.837331] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4441.769478] Lustre: Failing over lustre-MDT0000 [ 4442.085317] LustreError: 155310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4442.199789] Lustre: server umount lustre-MDT0000 complete [ 4442.746316] LustreError: 154196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.672160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.675966] LustreError: 153062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.692446] Lustre: Skipped 1 previous similar message [ 4445.704534] LustreError: 153062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4450.786226] LustreError: 154196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.827158] LustreError: 154196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4454.704526] LustreError: 153043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.58@tcp arrived at 1763406042 with bad export cookie 15390934016753165304 [ 4455.907410] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.966074] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4462.052111] Lustre: 152374:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406033/real 1763406033] req@000000003e0083f3 x1849065167938432/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763406049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4462.093468] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.983817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4467.170233] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4467.205687] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4472.288980] LustreError: 152370:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000063c813c5 x1849065167943936/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4472.614171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.681253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4476.984531] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4477.926406] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4477.967706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4479.335524] Lustre: lustre-MDT0000: Denying connection for new client 2b16c6fa-69cd-4658-8ac0-640e01b291ed (at 192.168.201.58@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4488.164627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.174714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.189148] Lustre: Skipped 2 previous similar messages [ 4493.282130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4493.297886] Lustre: Skipped 2 previous similar messages [ 4493.690112] LustreError: 156256:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4493.693313] LustreError: 156256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4493.812893] Lustre: server umount lustre-OST0000 complete [ 4498.171503] LustreError: 156459:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4498.183612] Lustre: 155812:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4498.188781] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4498.211147] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4498.220776] LustreError: 155812:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b740e965 x1849065167958144/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4498.231905] LustreError: 155812:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4498.239276] LustreError: 155812:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4498.246762] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 4498.402602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.434296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.462771] LustreError: 156459:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4498.484770] LustreError: 156459:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4498.685683] Lustre: server umount lustre-MDT0000 complete [ 4502.203978] LustreError: 156257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406089 with bad export cookie 15390934016753165402 [ 4502.212746] LustreError: 152370:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4502.223492] LustreError: 156257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4502.224256] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4504.383983] LustreError: 156660:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4504.395737] LustreError: 156660:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4504.621351] Lustre: server umount lustre-MDT0001 complete [ 4516.584264] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4519.606291] Key type lgssc unregistered [ 4519.918541] LNet: 157192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4519.927925] LNetError: 157192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4519.941144] LNet: Removed LNI 192.168.201.158@tcp [ 4520.921854] Key type .llcrypt unregistered [ 4520.926085] Key type ._llcrypt unregistered [ 4543.266130] Key type ._llcrypt registered [ 4543.268576] Key type .llcrypt registered [ 4543.386448] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4544.473461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4544.483829] alg: No test for adler32 (adler32-zlib) [ 4545.528616] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4545.776577] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4547.455838] Key type lgssc registered [ 4548.553483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4555.933757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4555.955839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.440394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.193869] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4567.010892] LustreError: 158586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.026975] LustreError: 158586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4568.890894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4569.198227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4573.327280] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4578.568811] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4582.148537] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4589.149380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4589.597179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.376234] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4600.980812] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4601.904359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4608.994786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4609.008505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.038251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4612.065313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4612.070220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4612.075061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.093831] Lustre: Skipped 1 previous similar message [ 4617.185634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.188729] Lustre: Skipped 1 previous similar message [ 4622.303455] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4622.307896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4622.363424] Lustre: Skipped 1 previous similar message [ 4622.416247] LustreError: 160833:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4622.617916] Lustre: server umount lustre-OST0000 complete [ 4627.431288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.435784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.458257] Lustre: Skipped 1 previous similar message [ 4627.474108] Lustre: Skipped 1 previous similar message [ 4632.716824] LustreError: 161036:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4632.727790] LustreError: 161036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4633.075449] Lustre: server umount lustre-MDT0000 complete [ 4637.328151] LustreError: 160207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406224 with bad export cookie 17778243834226070399 [ 4637.331024] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4637.345230] LustreError: 160207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4637.468528] LustreError: 161237:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4637.482451] LustreError: 161237:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4637.744516] Lustre: server umount lustre-MDT0001 complete [ 4648.199127] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4651.232344] Key type lgssc unregistered [ 4651.566494] LNet: 161769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.574429] LNetError: 161769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4651.597347] LNet: Removed LNI 192.168.201.158@tcp [ 4652.506226] Key type .llcrypt unregistered [ 4652.511380] Key type ._llcrypt unregistered [ 4681.932381] Key type ._llcrypt registered [ 4681.933986] Key type .llcrypt registered [ 4682.031878] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4683.386796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4683.446692] alg: No test for adler32 (adler32-zlib) [ 4684.650394] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4684.913287] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4686.623291] Key type lgssc registered [ 4687.635745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4696.530252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4696.556911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.362699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.810438] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4707.811878] LustreError: 163454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.834473] LustreError: 163454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4712.933892] LustreError: 163454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4713.519647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4713.832609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4717.818745] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4724.021567] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4727.362209] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4734.497305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4734.904287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.693667] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4743.098667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4747.096272] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4753.378706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.404557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4755.143982] LustreError: 165699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4755.292035] Lustre: server umount lustre-OST0000 complete [ 4760.034302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4760.049685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.059558] Lustre: Skipped 1 previous similar message [ 4760.066898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4760.069815] Lustre: Skipped 1 previous similar message [ 4763.622711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4763.623942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.756573] LustreError: 165900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4764.765791] LustreError: 165900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4764.880189] Lustre: server umount lustre-MDT0000 complete [ 4768.538425] LustreError: 165901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406355 with bad export cookie 12633888248212244494 [ 4768.553944] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.579547] LustreError: 165901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4768.770647] LustreError: 166103:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4768.775456] LustreError: 166103:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4769.039419] Lustre: server umount lustre-MDT0001 complete [ 4779.945948] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4782.891514] Key type lgssc unregistered [ 4783.247738] LNet: 166635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4783.263237] LNetError: 166635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4783.279781] LNet: Removed LNI 192.168.201.158@tcp [ 4784.062675] Key type .llcrypt unregistered [ 4784.064943] Key type ._llcrypt unregistered [ 4804.412029] Key type ._llcrypt registered [ 4804.413455] Key type .llcrypt registered [ 4804.503408] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4805.432757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4805.484255] alg: No test for adler32 (adler32-zlib) [ 4806.510895] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4806.725387] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4808.416168] Key type lgssc registered [ 4809.672066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4820.459356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4820.477494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.953583] Lustre: *** cfs_fail_loc=135, val=0*** [ 4822.163325] LustreError: 168139:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4822.172392] LustreError: 168139:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.158@tcp: cfg command failed: rc = -2 [ 4822.182661] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4822.182661] [ 4822.210984] LustreError: MGC192.168.201.158@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4822.246332] LustreError: 168100:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4822.259130] LustreError: 168100:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4822.282370] LustreError: 168100:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4822.309841] LustreError: 168100:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4822.481299] Lustre: server umount lustre-MDT0000 complete [ 4822.746265] LustreError: 168100:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4830.661929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.171497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4835.622623] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4841.448139] LustreError: 168512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4841.502438] LustreError: 168512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4845.651076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4846.083927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4849.955989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4855.308695] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4859.263213] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4866.529969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4866.534456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.564596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4867.044258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.862561] LustreError: 169856:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4867.871126] LustreError: 169856:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4867.990229] Lustre: server umount lustre-MDT0000 complete [ 4871.203748] LustreError: 168512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.229344] LustreError: 168512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4871.565053] LustreError: 168493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406458 with bad export cookie 6991598424152293272 [ 4871.568860] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4871.572843] LustreError: 168493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4871.733770] LustreError: 170058:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4871.740922] LustreError: 170058:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4872.043788] Lustre: server umount lustre-MDT0001 complete [ 4882.619782] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4885.924588] Key type lgssc unregistered [ 4886.306772] LNet: 170591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.319220] LNetError: 170591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4886.337374] LNet: Removed LNI 192.168.201.158@tcp [ 4887.370396] Key type .llcrypt unregistered [ 4887.372102] Key type ._llcrypt unregistered [ 4910.117768] Key type ._llcrypt registered [ 4910.130751] Key type .llcrypt registered [ 4910.297489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4910.572859] Key type .llcrypt unregistered [ 4910.577874] Key type ._llcrypt unregistered [ 4927.934847] Key type ._llcrypt registered [ 4927.938878] Key type .llcrypt registered [ 4928.092616] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4929.173220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4929.505334] alg: No test for adler32 (adler32-zlib) [ 4930.675865] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4930.930599] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4932.623167] Key type lgssc registered [ 4933.805793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4941.863879] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4941.890350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4957.535867] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4962.656115] LustreError: 172683:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 4989.346565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.470507] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5000.365639] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5007.459253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.153142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.244157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5012.767766] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5013.479286] LustreError: 172709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.492787] LustreError: 172709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5018.597534] LustreError: 172709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.718692] LustreError: 173608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5023.765792] LustreError: 173608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5023.927992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5024.667803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5030.088950] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5037.983416] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.643282] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5044.933881] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5059.553307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.558932] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.574487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.580983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5060.584788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.601718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.635045] Lustre: Skipped 1 previous similar message [ 5065.699910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5065.719096] Lustre: Skipped 1 previous similar message [ 5070.303142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5070.335113] LustreError: 175362:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5070.526778] Lustre: server umount lustre-OST0000 complete [ 5075.936476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5075.960390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5075.979115] Lustre: Skipped 1 previous similar message [ 5081.056964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.247292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5089.373860] LustreError: 175566:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5089.380807] LustreError: 175566:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5089.551229] Lustre: server umount lustre-MDT0000 complete [ 5091.296506] LustreError: 174827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.317975] LustreError: 174827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5094.016458] LustreError: 172690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406681 with bad export cookie 17160660953129824909 [ 5094.019314] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5094.039288] LustreError: 172690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5094.255250] LustreError: 175769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5094.262847] LustreError: 175769:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5094.543078] Lustre: server umount lustre-MDT0001 complete [ 5104.869507] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5107.783080] Key type lgssc unregistered [ 5108.088404] LNet: 176301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5108.101427] LNetError: 176301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5109.160621] LNet: Removed LNI 192.168.201.158@tcp [ 5110.004415] Key type .llcrypt unregistered [ 5110.005952] Key type ._llcrypt unregistered [ 5138.259274] Key type ._llcrypt registered [ 5138.263058] Key type .llcrypt registered [ 5138.430968] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5140.125900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5140.155125] alg: No test for adler32 (adler32-zlib) [ 5141.253954] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5141.585892] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5143.351275] Key type lgssc registered [ 5144.692193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5153.593666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5153.608310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5155.243527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.884857] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5165.537725] LustreError: 177698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.560459] LustreError: 177698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5168.908679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5169.296610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5173.617960] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5179.760278] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5183.329241] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5190.877688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5191.368573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.055322] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5199.616523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5205.552309] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5210.236437] Lustre: Failing over lustre-MDT0000 [ 5210.466027] LustreError: 179995:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5212.522356] LustreError: 179995:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5212.525372] LustreError: 179995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5212.610845] Lustre: server umount lustre-MDT0000 complete [ 5212.840547] LustreError: 178812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5215.205539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5215.209291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.215965] LustreError: 177698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.899730] LustreError: 177693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5217.929888] LustreError: 177693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5223.026751] LustreError: 177693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5223.050240] LustreError: 177693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5231.583299] Lustre: 177010:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763406802/real 1763406802] req@00000000f7339c8a x1849065973244928/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763406818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5231.645708] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5231.665940] LustreError: 178812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5231.676943] LustreError: 178812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5232.191180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.832657] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe2a21fdd45fd2e91 [ 5241.845791] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 5242.148355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.232120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5244.534147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5246.916604] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5247.476194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5247.520387] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5247.598156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5252.770380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5254.575861] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5267.940236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.944207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.945379] Lustre: Skipped 3 previous similar messages [ 5267.964617] Lustre: Skipped 1 previous similar message [ 5269.446240] LustreError: 181163:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5269.449073] LustreError: 181163:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5269.650825] Lustre: server umount lustre-OST0000 complete [ 5278.188464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.189704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5278.191252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5278.216629] Lustre: Skipped 1 previous similar message [ 5279.537250] LustreError: 181364:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5279.557161] LustreError: 181364:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5279.748760] Lustre: server umount lustre-MDT0000 complete [ 5283.304485] LustreError: 177699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.332928] LustreError: 177699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5284.184837] LustreError: 177680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406871 with bad export cookie 16330650234021293713 [ 5284.186841] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5284.215247] LustreError: 177680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5284.432454] LustreError: 181565:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5284.445373] LustreError: 181565:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5284.735979] Lustre: server umount lustre-MDT0001 complete [ 5295.303324] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5298.318358] Key type lgssc unregistered [ 5298.628553] LNet: 182096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5298.635118] LNetError: 182096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5298.647580] LNet: Removed LNI 192.168.201.158@tcp [ 5299.435929] Key type .llcrypt unregistered [ 5299.442959] Key type ._llcrypt unregistered [ 5321.618273] Key type ._llcrypt registered [ 5321.619685] Key type .llcrypt registered [ 5321.729265] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5322.355518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5322.368492] alg: No test for adler32 (adler32-zlib) [ 5323.448738] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5323.665706] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5325.343201] Key type lgssc registered [ 5327.109089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5335.211117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5335.238565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.032548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.613418] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5347.302547] LustreError: 183490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.309970] LustreError: 183490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5351.113798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5351.662804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5355.861175] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5361.366515] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5365.383445] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5372.922194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5373.510947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.657542] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5381.710758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5385.792295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5388.720185] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5394.219309] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5404.693270] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5422.561145] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5422.565942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5422.569092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5422.576976] Lustre: Skipped 1 previous similar message [ 5425.076257] LustreError: 185931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5425.185507] Lustre: server umount lustre-OST0000 complete [ 5432.802846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.814264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.816875] Lustre: Skipped 1 previous similar message [ 5437.929155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5437.938314] Lustre: Skipped 1 previous similar message [ 5443.048330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.062256] Lustre: Skipped 1 previous similar message [ 5444.067283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5444.223065] LustreError: 186133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5444.262598] LustreError: 186133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5444.373623] Lustre: server umount lustre-MDT0000 complete [ 5448.161967] LustreError: 183489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.191363] LustreError: 183489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5449.340089] LustreError: 185691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407036 with bad export cookie 3153869104671355449 [ 5449.348929] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5449.368655] LustreError: 185691:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5449.547797] LustreError: 186340:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5449.550943] LustreError: 186340:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5449.923931] Lustre: server umount lustre-MDT0001 complete [ 5461.653948] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5464.519412] Key type lgssc unregistered [ 5464.921105] LNet: 186872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5464.946636] LNetError: 186872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5464.970869] LNet: Removed LNI 192.168.201.158@tcp [ 5465.795141] Key type .llcrypt unregistered [ 5465.796893] Key type ._llcrypt unregistered [ 5492.158682] Key type ._llcrypt registered [ 5492.173277] Key type .llcrypt registered [ 5492.309912] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5492.975063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5493.031885] alg: No test for adler32 (adler32-zlib) [ 5494.927937] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5495.197045] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5496.978615] Key type lgssc registered [ 5498.541927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5508.183353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5508.223328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.192436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.131491] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5520.358995] LustreError: 188264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.381902] LustreError: 188264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5524.971476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5525.466945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.888379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5537.465551] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5541.299766] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5548.753528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5549.139064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.335192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5555.704473] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5563.904929] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5570.382439] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5580.808818] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5595.170416] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5601.084778] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5612.003297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.006102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.041653] LustreError: Skipped 1 previous similar message [ 5612.051055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5612.063051] Lustre: Skipped 1 previous similar message [ 5616.595867] LustreError: 191955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5616.915432] Lustre: server umount lustre-OST0000 complete [ 5622.242472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.249674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5622.285878] Lustre: Skipped 2 previous similar messages [ 5622.312176] Lustre: Skipped 1 previous similar message [ 5627.361620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5627.383514] Lustre: Skipped 1 previous similar message [ 5628.513852] LustreError: 192157:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5628.516556] LustreError: 192157:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5628.614677] Lustre: server umount lustre-MDT0000 complete [ 5632.274892] LustreError: 190964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407219 with bad export cookie 14514221394829451268 [ 5632.280846] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.285883] LustreError: 190964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5632.416192] LustreError: 192358:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5632.419551] LustreError: 192358:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5632.563474] Lustre: server umount lustre-MDT0001 complete [ 5643.065548] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5645.760724] Key type lgssc unregistered [ 5646.043681] LNet: 192889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5646.062943] LNetError: 192889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5646.081898] LNet: Removed LNI 192.168.201.158@tcp [ 5646.936438] Key type .llcrypt unregistered [ 5646.940285] Key type ._llcrypt unregistered [ 5669.205812] Key type ._llcrypt registered [ 5669.207963] Key type .llcrypt registered [ 5669.303460] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5670.232755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5670.470937] alg: No test for adler32 (adler32-zlib) [ 5671.528515] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5671.784767] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5673.519172] Key type lgssc registered [ 5674.548503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5683.813590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5683.850733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.639216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.568357] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5695.977445] LustreError: 194287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5696.014776] LustreError: 194287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5697.061746] LustreError: 194287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.894273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.118808] LustreError: 194287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.514993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5706.956653] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5713.272413] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5717.292944] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5724.576744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5725.171513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.568894] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5732.357158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5737.116111] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5752.801366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5752.804701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5752.813441] LustreError: Skipped 1 previous similar message [ 5752.819459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5752.822784] Lustre: Skipped 1 previous similar message [ 5756.991717] LustreError: 196685:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5757.063780] Lustre: server umount lustre-OST0000 complete [ 5763.045752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.048808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5763.087027] Lustre: Skipped 1 previous similar message [ 5763.109519] Lustre: Skipped 2 previous similar messages [ 5767.415985] LustreError: 196886:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5767.435760] LustreError: 196886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5767.655667] Lustre: server umount lustre-MDT0000 complete [ 5768.162541] LustreError: 194282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.170399] LustreError: 194282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5771.636919] LustreError: 194268:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407359 with bad export cookie 8354592336693024663 [ 5771.651565] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5771.779516] LustreError: 197087:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5771.783085] LustreError: 197087:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5771.974736] Lustre: server umount lustre-MDT0001 complete [ 5783.764587] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5786.565453] Key type lgssc unregistered [ 5786.923186] LNet: 197622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5786.936501] LNetError: 197622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5788.005745] LNet: Removed LNI 192.168.201.158@tcp [ 5789.028721] Key type .llcrypt unregistered [ 5789.030839] Key type ._llcrypt unregistered [ 5814.349246] Key type ._llcrypt registered [ 5814.351643] Key type .llcrypt registered [ 5814.451044] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5815.824123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5815.884572] alg: No test for adler32 (adler32-zlib) [ 5817.120231] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5817.480513] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5819.321326] Key type lgssc registered [ 5821.651172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5831.832450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5831.887068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.565584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.852753] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5842.926614] LustreError: 198870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.943224] LustreError: 198870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5848.052924] LustreError: 198870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.623428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5848.991140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5854.167471] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5860.300275] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5864.496643] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5872.704067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5873.242961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5878.397878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5880.761046] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5887.915809] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5902.309600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.331758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.337507] Lustre: Skipped 1 previous similar message [ 5906.858331] LustreError: 201269:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5906.968715] Lustre: server umount lustre-OST0000 complete [ 5912.544949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.548783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5912.559951] Lustre: Skipped 2 previous similar messages [ 5912.573476] Lustre: Skipped 1 previous similar message [ 5916.770684] LustreError: 201472:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5916.773491] LustreError: 201472:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5916.900382] Lustre: server umount lustre-MDT0000 complete [ 5917.664394] LustreError: 198870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.672027] LustreError: 198870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5920.124816] LustreError: 198849:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407507 with bad export cookie 12731166771367121371 [ 5920.137882] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5920.294549] LustreError: 201673:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5920.298625] LustreError: 201673:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5920.466883] Lustre: server umount lustre-MDT0001 complete [ 5931.019730] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5934.068795] Key type lgssc unregistered [ 5934.341513] LNet: 202208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5934.348954] LNetError: 202208:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5934.372911] LNet: Removed LNI 192.168.201.158@tcp [ 5935.299780] Key type .llcrypt unregistered [ 5935.301654] Key type ._llcrypt unregistered [ 5953.168718] Key type ._llcrypt registered [ 5953.170527] Key type .llcrypt registered [ 5953.306100] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5954.324331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5954.351697] alg: No test for adler32 (adler32-zlib) [ 5955.463286] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 5955.720099] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5957.375187] Key type lgssc registered [ 5959.061638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5968.244076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5968.295605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5969.874866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.758427] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5980.128788] LustreError: 203455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5980.158135] LustreError: 203455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5983.226353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5983.975836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5988.313889] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5993.770456] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5997.808324] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6006.357132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6006.986346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.805604] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6015.217092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6018.950539] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6029.052028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6029.157941] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6029.493273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6036.359164] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6043.844035] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6045.673363] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6057.153275] Lustre: Permanently deactivating lustre-OST0001 [ 6057.174847] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6062.692361] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6069.146286] Lustre: Permanently reactivating lustre-OST0001 [ 6069.174679] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6069.183237] Lustre: Skipped 2 previous similar messages [ 6069.321369] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.346502] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6069.352190] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6069.364615] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6069.399214] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6082.885130] LustreError: 206913:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6082.973385] Lustre: server umount lustre-OST0001 complete [ 6085.090288] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6085.093510] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.109547] LustreError: Skipped 1 previous similar message [ 6085.112151] LustreError: 205977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.150923] Lustre: Skipped 2 previous similar messages [ 6087.138940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6087.144599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6087.157238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.216177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.216647] LustreError: 205977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.220128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.233298] Lustre: Skipped 1 previous similar message [ 6090.261974] LustreError: 205977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6093.065085] LustreError: 207113:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6093.069641] LustreError: 207113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6093.169877] Lustre: server umount lustre-OST0000 complete [ 6100.454086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6100.507716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.520918] Lustre: Skipped 1 previous similar message [ 6103.303205] LustreError: 207315:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6103.311388] LustreError: 207315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6103.647993] Lustre: server umount lustre-MDT0000 complete [ 6105.570081] LustreError: 204619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.591349] LustreError: 204619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6108.576908] LustreError: 203435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407696 with bad export cookie 3022785400826174782 [ 6108.587773] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.610136] LustreError: 203435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6109.007894] LustreError: 207516:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6109.035389] LustreError: 207516:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6109.378349] Lustre: server umount lustre-MDT0001 complete [ 6121.665976] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6125.179839] Key type lgssc unregistered [ 6125.553159] LNet: 208049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6125.564097] LNetError: 208049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6126.639409] LNet: Removed LNI 192.168.201.158@tcp [ 6127.453953] Key type .llcrypt unregistered [ 6127.457224] Key type ._llcrypt unregistered [ 6156.081687] Key type ._llcrypt registered [ 6156.083398] Key type .llcrypt registered [ 6156.241888] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6168.607680] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6169.622405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6169.641711] alg: No test for adler32 (adler32-zlib) [ 6170.750973] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6171.017406] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6172.711232] Key type lgssc registered [ 6173.768419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6184.706557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6193.827582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6200.492332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6208.145754] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6216.113777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6216.166111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6216.175839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.466872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6217.491988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6217.581681] Lustre: lustre-MDT0000: new disk, initializing [ 6217.744363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.777326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6221.496890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6230.899369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6230.993808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6231.086637] Lustre: 211977:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6231.148493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6231.156318] Lustre: Skipped 1 previous similar message [ 6231.261936] Lustre: lustre-MDT0001: new disk, initializing [ 6231.355254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6231.398995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6231.407973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6234.495277] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6242.420839] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6247.627522] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6254.503610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.601908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6254.785827] Lustre: lustre-OST0000: new disk, initializing [ 6254.793398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6254.896501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6260.209812] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6261.322192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6261.333495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6261.451329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6267.357663] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6272.210051] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6272.470632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6278.242491] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6278.514920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6282.213470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.217101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.225576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.358450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6287.361181] Lustre: Skipped 1 previous similar message [ 6287.578301] LustreError: 214914:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6287.881396] Lustre: server umount lustre-OST0000 complete [ 6297.576867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.579651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.582284] Lustre: Skipped 2 previous similar messages [ 6297.599223] Lustre: Skipped 2 previous similar messages [ 6301.337223] LustreError: 215115:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6301.342141] LustreError: 215115:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6301.554387] Lustre: server umount lustre-MDT0000 complete [ 6306.115146] LustreError: 211968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407893 with bad export cookie 1124134172728382745 [ 6306.123441] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.207798] LustreError: 211968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6306.425526] LustreError: 215318:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6306.429606] LustreError: 215318:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6306.582292] Lustre: server umount lustre-MDT0001 complete [ 6315.112574] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 14:31:40 (1763407900) [ 6321.430090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.866327] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.889780] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6321.961740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6325.829655] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6327.270859] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.405705] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6335.667663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6336.040320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.357621] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6347.993749] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6352.227746] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6357.572358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6357.887611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6362.581146] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6368.159199] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6370.569068] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6376.930686] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6385.647424] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6391.011823] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6405.518844] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6405.551535] Lustre: Skipped 1 previous similar message [ 6425.256343] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6425.261376] Lustre: Skipped 2 previous similar messages [ 6462.196683] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6462.218165] Lustre: Skipped 4 previous similar messages [ 6527.383931] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6527.390261] Lustre: Skipped 7 previous similar messages [ 6535.137718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.142535] LustreError: Skipped 1 previous similar message [ 6535.145370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.153817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.161053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6540.550204] LustreError: 219461:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6540.565602] LustreError: 219461:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6540.791594] Lustre: server umount lustre-OST0000 complete [ 6545.888618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6545.899225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6545.920124] Lustre: Skipped 1 previous similar message [ 6545.929717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6545.932285] Lustre: Skipped 2 previous similar messages [ 6556.640487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6556.642708] Lustre: Skipped 4 previous similar messages [ 6559.199160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6559.322539] LustreError: 219662:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6559.324750] LustreError: 219662:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6559.451766] Lustre: server umount lustre-MDT0000 complete [ 6560.741377] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.907094] LustreError: 215970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408151 with bad export cookie 1124134172728383627 [ 6563.923975] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6563.933990] LustreError: 215970:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6564.581468] Lustre: server umount lustre-MDT0001 complete [ 6576.893855] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6580.292759] Key type lgssc unregistered [ 6580.582980] LNet: 220399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6580.594667] LNetError: 220399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6580.611936] LNet: Removed LNI 192.168.201.158@tcp [ 6581.826766] Key type .llcrypt unregistered [ 6581.828404] Key type ._llcrypt unregistered [ 6609.369462] Key type ._llcrypt registered [ 6609.377764] Key type .llcrypt registered [ 6609.536293] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6610.336494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6610.360256] alg: No test for adler32 (adler32-zlib) [ 6611.502713] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6611.764883] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6613.431215] Key type lgssc registered [ 6614.824253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6623.614436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6623.647065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.185475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6629.306766] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6635.492219] LustreError: 221794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.507914] LustreError: 221794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6638.142538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6638.537428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6642.764729] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6647.963116] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6651.348979] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6658.141206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6658.465747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6664.770819] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6670.987981] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6687.817926] LustreError: 224200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6688.049781] Lustre: server umount lustre-OST0000 complete [ 6694.889868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6694.901976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.928803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.532148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.553117] Lustre: Skipped 3 previous similar messages [ 6703.583823] Lustre: 221106:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763408275/real 1763408275] req@0000000004ec7923 x1849067515708800/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763408291 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6703.591233] LustreError: 224402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6703.629287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6703.629294] Lustre: Skipped 1 previous similar message [ 6703.633885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6703.664870] LustreError: 224402:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6703.800384] Lustre: Skipped 1 previous similar message [ 6703.950246] Lustre: server umount lustre-MDT0000 complete [ 6705.695135] Lustre: 221106:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763408277/real 1763408277] req@00000000f7ecc43b x1849067515709824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763408293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6708.704184] LustreError: 221791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.748940] LustreError: 221791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6708.923506] LustreError: 224043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408296 with bad export cookie 9030555811155677805 [ 6708.926546] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6708.955800] LustreError: 224043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6709.249318] LustreError: 224604:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6709.297685] LustreError: 224604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6709.892736] Lustre: server umount lustre-MDT0001 complete [ 6724.652943] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6727.534257] Key type lgssc unregistered [ 6727.919265] LNet: 225136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6727.926409] LNetError: 225136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6727.958284] LNet: Removed LNI 192.168.201.158@tcp [ 6729.119987] Key type .llcrypt unregistered [ 6729.122985] Key type ._llcrypt unregistered [ 6758.217072] Key type ._llcrypt registered [ 6758.223334] Key type .llcrypt registered [ 6758.305936] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6758.583288] Key type .llcrypt unregistered [ 6758.585072] Key type ._llcrypt unregistered [ 6786.151144] Key type ._llcrypt registered [ 6786.152985] Key type .llcrypt registered [ 6786.247202] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6802.274806] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6803.643365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6803.697806] alg: No test for adler32 (adler32-zlib) [ 6804.732976] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6804.973977] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6806.658371] Key type lgssc registered [ 6807.772089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6818.289758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6825.505610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6831.350569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6838.048275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6840.173226] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6551 sec ========== 14:40:25 (1763408425) [ 6842.209121] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:40:27 (1763408427) === [ 6845.856555] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:40:31 (1763408431) === [ 6893.667927] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 6898.383987] Key type lgssc unregistered [ 6898.857683] LNet: 231252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.882744] LNetError: 231252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6898.908810] LNet: Removed LNI 192.168.201.158@tcp [ 6899.993658] Key type .llcrypt unregistered [ 6899.999076] Key type ._llcrypt unregistered