[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 527152871 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003058] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008011] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011135] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014065] *** VALIDATE selinux *** [ 0.022637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026774] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] *** VALIDATE tmpfs *** [ 0.032381] *** VALIDATE proc *** [ 0.034120] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037221] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043729] debug: unmapping init [mem 0xffffffff8b059000-0xffffffff8b060fff] [ 0.045156] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046630] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049024] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054285] rcu: Hierarchical SRCU implementation. [ 0.056390] smp: Bringing up secondary CPUs ... [ 0.057560] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.074473] smp: Brought up 1 node, 4 CPUs [ 0.076015] smpboot: Max logical packages: 1 [ 0.077032] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.249543] node 0 deferred pages initialised in 168ms [ 0.253144] devtmpfs: initialized [ 0.254389] x86/mm: Memory block size: 128MB [ 0.258166] gcov: version magic: 0x41383552 [ 0.262548] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.266098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.271346] pinctrl core: initialized pinctrl subsystem [ 0.273186] [ 0.273784] ************************************************************* [ 0.276014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.279014] ** ** [ 0.281013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.284017] ** ** [ 0.286012] ** This means that this kernel is built to expose internal ** [ 0.289016] ** IOMMU data structures, which may compromise security on ** [ 0.291014] ** your system. ** [ 0.294017] ** ** [ 0.296014] ** If you see this message and you are not debugging the ** [ 0.299017] ** kernel, report this immediately to your vendor! ** [ 0.301017] ** ** [ 0.304017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.306013] ************************************************************* [ 0.308808] NET: Registered protocol family 16 [ 0.311445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.314070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.321015] cpuidle: using governor menu [ 0.322824] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325458] PCI: Using configuration type 1 for base access [ 0.327119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.336136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.342352] cryptd: max_cpu_qlen set to 1000 [ 0.345032] ACPI: Added _OSI(Module Device) [ 0.346019] ACPI: Added _OSI(Processor Device) [ 0.348017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.349011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.352535] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.359288] ACPI: Interpreter enabled [ 0.360056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362013] ACPI: Using IOAPIC for interrupt routing [ 0.363074] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.368437] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.378258] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.380069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.383030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.387115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.392380] acpiphp: Slot [2] registered [ 0.394242] acpiphp: Slot [3] registered [ 0.396110] acpiphp: Slot [4] registered [ 0.397118] acpiphp: Slot [5] registered [ 0.398107] acpiphp: Slot [6] registered [ 0.399105] acpiphp: Slot [7] registered [ 0.400078] acpiphp: Slot [8] registered [ 0.401080] acpiphp: Slot [9] registered [ 0.403109] acpiphp: Slot [10] registered [ 0.404103] acpiphp: Slot [11] registered [ 0.406121] acpiphp: Slot [12] registered [ 0.407071] acpiphp: Slot [13] registered [ 0.408072] acpiphp: Slot [14] registered [ 0.409117] acpiphp: Slot [15] registered [ 0.411135] acpiphp: Slot [16] registered [ 0.413101] acpiphp: Slot [17] registered [ 0.414123] acpiphp: Slot [18] registered [ 0.416116] acpiphp: Slot [19] registered [ 0.417095] acpiphp: Slot [20] registered [ 0.418130] acpiphp: Slot [21] registered [ 0.420108] acpiphp: Slot [22] registered [ 0.422120] acpiphp: Slot [23] registered [ 0.423095] acpiphp: Slot [24] registered [ 0.424114] acpiphp: Slot [25] registered [ 0.426101] acpiphp: Slot [26] registered [ 0.427099] acpiphp: Slot [27] registered [ 0.429135] acpiphp: Slot [28] registered [ 0.430095] acpiphp: Slot [29] registered [ 0.432101] acpiphp: Slot [30] registered [ 0.433111] acpiphp: Slot [31] registered [ 0.435080] PCI host bridge to bus 0000:00 [ 0.436022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.438041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.439000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.441047] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444037] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.446108] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.450246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460927] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.471767] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.476000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478035] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.480026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.483028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.486518] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.488763] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.492046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494621] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.501029] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.515025] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.520831] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.525000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.539028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556041] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.563000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.576019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.594023] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.601000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.607026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.613023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.634045] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.644528] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.655018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.660023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.674022] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.682200] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.689036] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.697026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.712023] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.721889] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.726017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.732024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.743019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.752000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.753429] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.756372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.758351] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.760239] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.766150] iommu: Default domain type: Passthrough [ 0.768439] SCSI subsystem initialized [ 0.769159] ACPI: bus type USB registered [ 0.771209] usbcore: registered new interface driver usbfs [ 0.773124] usbcore: registered new interface driver hub [ 0.775140] usbcore: registered new device driver usb [ 0.777254] pps_core: LinuxPPS API ver. 1 registered [ 0.779019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782121] PTP clock support registered [ 0.784204] EDAC MC: Ver: 3.0.0 [ 0.786565] PCI: Using ACPI for IRQ routing [ 0.788884] NetLabel: Initializing [ 0.792021] NetLabel: domain hash size = 128 [ 0.793012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.795097] NetLabel: unlabeled traffic allowed by default [ 0.797151] vgaarb: loaded [ 0.798377] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.800019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.807022] clocksource: Switched to clocksource kvm-clock [ 0.916668] VFS: Disk quotas dquot_6.6.0 [ 0.918148] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.920708] *** VALIDATE ramfs *** [ 0.922114] *** VALIDATE hugetlbfs *** [ 0.924036] pnp: PnP ACPI init [ 0.926495] pnp: PnP ACPI: found 6 devices [ 0.948159] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.951623] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.953840] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.955994] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.958458] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.961024] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.963995] NET: Registered protocol family 2 [ 0.966408] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.971153] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.974176] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.978805] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.981652] TCP: Hash tables configured (established 65536 bind 65536) [ 0.984104] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986849] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989128] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991597] NET: Registered protocol family 1 [ 0.996839] RPC: Registered named UNIX socket transport module. [ 0.999060] RPC: Registered udp transport module. [ 1.001122] RPC: Registered tcp transport module. [ 1.002672] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005905] NET: Registered protocol family 44 [ 1.008152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.010530] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.013673] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015913] PCI: CLS 0 bytes, default 64 [ 1.017419] Unpacking initramfs... [ 2.438537] debug: unmapping init [mem 0xffff91e53cc54000-0xffff91e53ffbffff] [ 2.442438] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444732] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447545] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.952369] Initialise system trusted keyrings [ 2.954319] Key type blacklist registered [ 2.956602] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.965858] zbud: loaded [ 2.969475] *** VALIDATE nfs *** [ 2.970938] *** VALIDATE nfs4 *** [ 2.972953] pstore: using deflate compression [ 2.976812] Platform Keyring initialized [ 3.096232] NET: Registered protocol family 38 [ 3.098613] Key type asymmetric registered [ 3.100272] Asymmetric key parser 'x509' registered [ 3.102405] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.105758] io scheduler mq-deadline registered [ 3.107649] io scheduler kyber registered [ 3.109981] io scheduler bfq registered [ 3.112118] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.115477] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.118103] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.121346] ACPI: Power Button [PWRF] [ 3.214645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.307826] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.500788] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.599064] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.790959] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.825216] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.857677] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.864389] Non-volatile memory driver v1.3 [ 3.865983] Linux agpgart interface v0.103 [ 3.912263] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.916256] vda: detected capacity change from 0 to 68526080 [ 3.933230] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.935868] vdb: detected capacity change from 0 to 1073741824 [ 3.950884] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.953870] vdc: detected capacity change from 0 to 2621440000 [ 3.967263] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.970455] vdd: detected capacity change from 0 to 2621440000 [ 3.985103] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.988029] vde: detected capacity change from 0 to 4294967296 [ 4.004872] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.007897] vdf: detected capacity change from 0 to 4294967296 [ 4.017138] libphy: Fixed MDIO Bus: probed [ 4.024311] usbcore: registered new interface driver usbserial_generic [ 4.026863] usbserial: USB Serial support registered for generic [ 4.029306] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.034092] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.036095] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.041093] mousedev: PS/2 mouse device common for all mice [ 4.045197] rtc_cmos 00:05: RTC can wake from S4 [ 4.047752] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.050616] rtc_cmos 00:05: registered as rtc0 [ 4.053374] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.058961] intel_pstate: CPU model not supported [ 4.063464] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.068653] hid: raw HID events driver (C) Jiri Kosina [ 4.071871] usbcore: registered new interface driver usbhid [ 4.075334] usbhid: USB HID core driver [ 4.077377] drop_monitor: Initializing network drop monitor service [ 4.081397] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.085738] Initializing XFRM netlink socket [ 4.087886] NET: Registered protocol family 10 [ 4.091049] Segment Routing with IPv6 [ 4.092504] NET: Registered protocol family 17 [ 4.094573] mpls_gso: MPLS GSO support [ 4.099404] RAS: Correctable Errors collector initialized. [ 4.101646] AVX version of gcm_enc/dec engaged. [ 4.103489] AES CTR mode by8 optimization enabled [ 4.190554] sched_clock: Marking stable (4190527992, 0)->(5197520822, -1006992830) [ 4.194553] registered taskstats version 1 [ 4.196566] Loading compiled-in X.509 certificates [ 4.198833] zswap: loaded using pool lzo/zbud [ 4.224362] Key type big_key registered [ 4.236167] Key type encrypted registered [ 4.237864] ima: No TPM chip found, activating TPM-bypass! [ 4.239867] ima: Allocated hash algorithm: sha1 [ 4.241295] ima: No architecture policies found [ 4.242644] evm: Initialising EVM extended attributes: [ 4.244338] evm: security.selinux [ 4.245487] evm: security.ima [ 4.246544] evm: security.capability [ 4.247868] evm: HMAC attrs: 0x1 [ 4.250432] rtc_cmos 00:05: setting system clock to 2025-11-17 01:09:24 UTC (1763341764) [ 4.257695] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 4.260707] debug: unmapping init [mem 0xffffffff8ad82000-0xffffffff8b058fff] [ 4.270078] Write protecting the kernel read-only data: 28672k [ 4.273352] debug: unmapping init [mem 0xffffffff89403000-0xffffffff895fffff] [ 4.276057] debug: unmapping init [mem 0xffffffff89d14000-0xffffffff89dfffff] [ 4.324445] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.332847] systemd[1]: Detected virtualization kvm. [ 4.334690] systemd[1]: Detected architecture x86-64. [ 4.336654] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.362661] systemd[1]: No hostname configured. [ 4.364151] systemd[1]: Set hostname to . [ 4.365776] random: systemd: uninitialized urandom read (16 bytes read) [ 4.367709] systemd[1]: Initializing machine ID from random generator. [ 4.517532] random: systemd: uninitialized urandom read (16 bytes read) [ 4.520414] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.525695] random: systemd: uninitialized urandom read (16 bytes read) [ 4.528172] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.533511] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.153519] device-mapper: uevent: version 1.0.3 [ 5.155836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.959424] random: fast init done [ 5.990353] scsi host0: ata_piix [ 6.008406] virtio_net virtio0 ens2: renamed from eth0 [ 6.022563] scsi host1: ata_piix [ 6.024435] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.026949] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.854671] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.752856] random: crng init done [ 10.754345] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.302334] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.603421] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.900843] SELinux: Disabled at runtime. [ 12.967779] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.976837] systemd[1]: Detected virtualization kvm. [ 12.978953] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.526325] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.529866] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.535862] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.541989] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.545486] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.557407] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.566924] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 13.645948] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 14.158962] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.480404] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.583770] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.771644] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.792372] EDAC sbridge: Ver: 1.1.2 [ 16.533814] Key type dns_resolver registered [ 16.844528] NFS: Registering the id_resolver key type [ 16.846720] Key type id_resolver registered [ 16.848859] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-server login: [ 44.040554] libcfs: loading out-of-tree module taints kernel. [ 44.052889] Key type ._llcrypt registered [ 44.054436] Key type .llcrypt registered [ 44.103675] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 58.509865] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 60.277714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.300118] alg: No test for adler32 (adler32-zlib) [ 61.826347] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 62.889677] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 64.735264] Key type lgssc registered [ 67.005473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.352686] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 88.786806] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.536883] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.622504] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.415036] hrtimer: interrupt took 2535479 ns [ 111.755585] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 127.770357] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 142.139350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 142.207223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.224601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.526742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.574841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.652852] Lustre: lustre-MDT0000: new disk, initializing [ 143.737266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.758687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.694848] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.901860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.975272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 162.056951] Lustre: 6531:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 162.089658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 162.093417] Lustre: Skipped 1 previous similar message [ 162.155687] Lustre: lustre-MDT0001: new disk, initializing [ 162.238334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 162.261294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 162.272820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 166.732130] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.743946] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 181.591626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 181.715045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 181.922794] Lustre: lustre-OST0000: new disk, initializing [ 181.926808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.998735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.337811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.342239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.396523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.656094] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.743866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.828564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 200.904849] Lustre: lustre-OST0001: new disk, initializing [ 200.912836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.978732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.659756] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.944150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.952099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.011328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 217.473192] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.600865] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 234.661828] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.800963] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 243.686884] Lustre: DEBUG MARKER: Client: 2.16.61.48 [ 246.176209] Lustre: DEBUG MARKER: MDS: 2.16.61.48 [ 248.827802] Lustre: DEBUG MARKER: OSS: 2.16.61.48 [ 250.547275] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:13:29 EST 2025 [ 268.440317] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 269.801233] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.152538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.172282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.208370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.689662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.692260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.176509] LustreError: 12239:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 291.289612] Lustre: server umount lustre-MDT0000 complete [ 294.983177] LustreError: 6521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342055 with bad export cookie 17235841447426171531 [ 294.999064] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.011588] LustreError: 6521:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.928301] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.930175] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.930657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.930662] Lustre: Skipped 3 previous similar messages [ 296.954352] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 296.959594] Lustre: Skipped 3 previous similar messages [ 299.936858] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.937956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 299.990077] Lustre: Skipped 2 previous similar messages [ 301.275680] LustreError: 12441:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 301.283464] LustreError: 12441:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 301.473569] Lustre: server umount lustre-MDT0001 complete [ 305.018260] LustreError: 12642:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 305.026272] LustreError: 12642:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 305.075591] Lustre: server umount lustre-OST0000 complete [ 308.387106] LustreError: 12842:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 308.399415] LustreError: 12842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 308.483242] Lustre: server umount lustre-OST0001 complete [ 314.072544] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 322.115361] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 330.768231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.094086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.858677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.748739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.102499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.176316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.341405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 354.375416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 354.483696] Lustre: lustre-MDT0000: new disk, initializing [ 354.594860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.622578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 358.419128] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 368.849867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 368.954954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 369.085882] Lustre: 15028:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 369.135813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 369.138740] Lustre: Skipped 1 previous similar message [ 369.191052] Lustre: lustre-MDT0001: new disk, initializing [ 369.285371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 369.329091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 369.352957] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 373.198602] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 379.944262] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 383.159917] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 388.868167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 388.920018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 389.156665] Lustre: lustre-OST0000: new disk, initializing [ 389.163078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 389.241381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.232879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 391.237687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 391.405716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 394.252379] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 400.612224] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 404.363506] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.561499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 408.406685] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 408.607334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 412.128189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 412.130153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.148211] LustreError: Skipped 1 previous similar message [ 412.167095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 412.171032] Lustre: Skipped 1 previous similar message [ 416.441782] LustreError: 17967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 416.455908] LustreError: 17967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 416.619732] Lustre: server umount lustre-OST0000 complete [ 420.833194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.836660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.841961] Lustre: Skipped 1 previous similar message [ 420.848710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 426.321570] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 426.326789] LustreError: 18168:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 426.436784] Lustre: server umount lustre-MDT0000 complete [ 427.489260] LustreError: 15035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.504326] LustreError: 15035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 429.828387] LustreError: 15019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342190 with bad export cookie 17235841447426173064 [ 429.829296] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.854717] LustreError: 15019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 430.181842] Lustre: server umount lustre-MDT0001 complete [ 433.675612] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:16:33 (1763342193) [ 439.736871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.192382] LustreError: 18895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.274904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.320210] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 445.412509] LustreError: 18895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 451.439309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 455.407791] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 460.065717] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 462.871873] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 469.012663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 474.117633] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 478.825771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 496.101080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.106515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.112447] Lustre: Skipped 1 previous similar message [ 496.133732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.136566] Lustre: Skipped 2 previous similar messages [ 502.148589] LustreError: 21159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 502.156105] LustreError: 21159:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 502.327476] Lustre: server umount lustre-OST0000 complete [ 520.160055] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 520.427842] Lustre: server umount lustre-MDT0000 complete [ 521.700961] LustreError: 20106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.741342] LustreError: 20106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 523.912742] LustreError: 18874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342284 with bad export cookie 17235841447426173946 [ 523.923741] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.930308] LustreError: 18874:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 524.273746] Lustre: server umount lustre-MDT0001 complete [ 534.512319] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 537.141685] Key type lgssc unregistered [ 537.486538] LNet: 22095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.504090] LNetError: 22095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.526720] LNet: Removed LNI 192.168.204.148@tcp [ 538.397566] Key type .llcrypt unregistered [ 538.399281] Key type ._llcrypt unregistered [ 560.231249] Key type ._llcrypt registered [ 560.232700] Key type .llcrypt registered [ 560.342892] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 561.499659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.666672] alg: No test for adler32 (adler32-zlib) [ 562.788448] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 563.162404] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 564.991256] Key type lgssc registered [ 566.125419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.806511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.824382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.436136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.430756] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 585.705363] LustreError: 23486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.715624] LustreError: 23486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 586.722180] LustreError: 23486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.425348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 587.710474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 591.380722] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 596.475569] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.447325] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 608.207652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 608.763174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.442659] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 621.138284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 621.480635] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 637.418485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.433935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.458460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 641.515573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 641.523142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.535455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 642.677391] LustreError: 26174:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 642.788726] Lustre: server umount lustre-OST0000 complete [ 646.625861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.627169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.637593] Lustre: Skipped 1 previous similar message [ 646.642641] Lustre: Skipped 1 previous similar message [ 651.748567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 651.763579] Lustre: Skipped 2 previous similar messages [ 652.583249] LustreError: 26376:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 652.585868] LustreError: 26376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 652.763613] Lustre: server umount lustre-MDT0000 complete [ 656.203931] LustreError: 23466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342416 with bad export cookie 4205355623850936378 [ 656.204747] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.216457] LustreError: 23466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 656.346565] LustreError: 26577:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 656.352856] LustreError: 26577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 656.485979] Lustre: server umount lustre-MDT0001 complete [ 664.877183] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 667.520774] Key type lgssc unregistered [ 667.801743] LNet: 27109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.814300] LNetError: 27109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.848278] LNet: Removed LNI 192.168.204.148@tcp [ 668.630165] Key type .llcrypt unregistered [ 668.632242] Key type ._llcrypt unregistered [ 688.581836] Key type ._llcrypt registered [ 688.584843] Key type .llcrypt registered [ 688.679573] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 689.600169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 689.702948] alg: No test for adler32 (adler32-zlib) [ 690.744205] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 690.936751] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 692.623300] Key type lgssc registered [ 693.703791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 700.543295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.569158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.047576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.412753] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 712.161028] LustreError: 28484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.171550] LustreError: 28484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 712.744779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 713.044354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 716.233285] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 720.526177] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 723.344338] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 733.922126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 734.191448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 738.778874] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 743.692509] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 747.569959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 757.728284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 757.738621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.745392] Lustre: Skipped 1 previous similar message [ 757.750669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 757.765707] Lustre: Skipped 1 previous similar message [ 759.865283] LustreError: 31076:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 760.108449] Lustre: server umount lustre-OST0000 complete [ 764.385388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 764.395489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.408035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 767.969725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 767.978059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 768.029101] Lustre: Skipped 1 previous similar message [ 773.093686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.105938] Lustre: Skipped 1 previous similar message [ 778.208096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 778.211228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.223137] Lustre: Skipped 1 previous similar message [ 778.313759] LustreError: 31278:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 778.318655] LustreError: 31278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 778.384660] Lustre: server umount lustre-MDT0000 complete [ 781.615804] LustreError: 28465:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342541 with bad export cookie 4480001833578805721 [ 781.624161] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.832848] LustreError: 31480:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 781.840938] LustreError: 31480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 782.049914] Lustre: server umount lustre-MDT0001 complete [ 790.356668] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 792.833737] Key type lgssc unregistered [ 793.115997] LNet: 32014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.123991] LNetError: 32014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.135044] LNet: Removed LNI 192.168.204.148@tcp [ 793.762207] Key type .llcrypt unregistered [ 793.765480] Key type ._llcrypt unregistered [ 812.285511] Key type ._llcrypt registered [ 812.289523] Key type .llcrypt registered [ 812.367179] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 813.041176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 813.056631] alg: No test for adler32 (adler32-zlib) [ 814.058530] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 814.210233] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 815.847619] Key type lgssc registered [ 816.834921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 823.640294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 823.678188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.260782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.588168] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 835.576311] LustreError: 33386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.598137] LustreError: 33386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 838.961716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 839.255962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.211595] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 848.332984] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.821623] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 857.537653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.943701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.787948] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 868.285630] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 870.267049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 879.585362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.590149] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.602676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.607885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.616092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.622182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.625024] Lustre: Skipped 1 previous similar message [ 885.696487] LustreError: 35633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 885.728171] LustreError: 35544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.733874] LustreError: 35544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 885.789642] Lustre: server umount lustre-OST0000 complete [ 890.337509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.342371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.349647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.969032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.981557] Lustre: Skipped 2 previous similar messages [ 901.088466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.091396] Lustre: Skipped 1 previous similar message [ 903.135320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 903.227668] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 903.239831] LustreError: 35834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 903.353409] Lustre: server umount lustre-MDT0000 complete [ 906.223338] LustreError: 33386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.250604] LustreError: 33386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 906.774369] LustreError: 33366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342667 with bad export cookie 6883992159173803647 [ 906.782354] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 906.799182] LustreError: 33366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 906.947464] LustreError: 36039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 906.956231] LustreError: 36039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 907.202704] Lustre: server umount lustre-MDT0001 complete [ 916.415795] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 918.924198] Key type lgssc unregistered [ 919.203202] LNet: 36572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.211991] LNetError: 36572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.233687] LNet: Removed LNI 192.168.204.148@tcp [ 919.982538] Key type .llcrypt unregistered [ 919.984686] Key type ._llcrypt unregistered [ 940.641167] Key type ._llcrypt registered [ 940.642591] Key type .llcrypt registered [ 940.729544] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 941.687416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.781567] alg: No test for adler32 (adler32-zlib) [ 942.864364] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 943.134593] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 944.802638] Key type lgssc registered [ 945.765567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.874358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.895377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 954.586846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.998208] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 964.070459] LustreError: 37967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 964.093264] LustreError: 37967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 965.338345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 965.615788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 968.984434] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 973.458994] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 976.505738] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.969512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 982.189312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.665788] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 991.009669] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 999.617933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1006.880240] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1010.151530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.156973] Lustre: Skipped 1 previous similar message [ 1010.157455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.158699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.174503] Lustre: Skipped 1 previous similar message [ 1011.970506] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1015.265026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.385167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1020.389772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1020.396417] Lustre: Skipped 3 previous similar messages [ 1020.429106] LustreError: 40235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1020.538767] Lustre: server umount lustre-OST0000 complete [ 1040.864774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.866495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1040.883200] Lustre: Skipped 1 previous similar message [ 1040.903308] Lustre: Skipped 1 previous similar message [ 1044.261836] LustreError: 40612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1044.272982] LustreError: 40612:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1044.431040] Lustre: server umount lustre-MDT0000 complete [ 1045.984396] LustreError: 38689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.013145] LustreError: 38689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1047.427920] LustreError: 37947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342807 with bad export cookie 6079114437043261706 [ 1047.434692] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.612637] LustreError: 40814:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1053.616117] LustreError: 40814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1053.727269] Lustre: server umount lustre-MDT0001 complete [ 1061.452235] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1063.592882] Key type lgssc unregistered [ 1063.819947] LNet: 41345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.842133] LNetError: 41345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.869845] LNet: Removed LNI 192.168.204.148@tcp [ 1065.628606] Key type .llcrypt unregistered [ 1065.631576] Key type ._llcrypt unregistered [ 1084.359754] Key type ._llcrypt registered [ 1084.361438] Key type .llcrypt registered [ 1084.475185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1085.488647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.561712] alg: No test for adler32 (adler32-zlib) [ 1086.581668] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1086.813234] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1088.503166] Key type lgssc registered [ 1089.707120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1096.294466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1096.314621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.651993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.866820] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1106.914505] LustreError: 42718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.945417] LustreError: 42718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1107.532239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1107.778565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1110.790407] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1115.138318] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1118.360839] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1125.252205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1125.554938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.539669] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1137.396287] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1141.996706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1147.368656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.370245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.380115] Lustre: Skipped 2 previous similar messages [ 1147.391357] Lustre: Skipped 1 previous similar message [ 1148.186705] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1149.228170] LustreError: 44962:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1149.408378] Lustre: server umount lustre-MDT0000 complete [ 1152.485391] LustreError: 42718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.493378] LustreError: 42718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1152.591337] LustreError: 42698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342912 with bad export cookie 7455988572223893428 [ 1152.595493] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.617108] LustreError: 42698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1152.777958] LustreError: 45163:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1152.782539] LustreError: 45163:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1152.891297] Lustre: server umount lustre-MDT0001 complete [ 1164.886508] LustreError: 45164:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.48@tcp arrived at 1763342925 with bad export cookie 7455988572223894093 [ 1183.418830] LustreError: 45419:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1183.424291] LustreError: 45419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1183.582431] Lustre: server umount lustre-OST0000 complete [ 1196.581097] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1198.725442] Key type lgssc unregistered [ 1199.009282] LNet: 46259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.013800] LNetError: 46259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.035666] LNet: Removed LNI 192.168.204.148@tcp [ 1199.853269] Key type .llcrypt unregistered [ 1199.857358] Key type ._llcrypt unregistered [ 1218.359110] Key type ._llcrypt registered [ 1218.363559] Key type .llcrypt registered [ 1218.448220] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1219.282460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.326544] alg: No test for adler32 (adler32-zlib) [ 1220.457902] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1220.640820] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1222.327195] Key type lgssc registered [ 1223.246621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1229.299292] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1229.317828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1244.831358] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1249.951209] LustreError: 47604:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 1276.639863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.154427] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1287.494955] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1451.162090] LustreError: 48311:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1451.312301] Lustre: server umount lustre-OST0000 complete [ 1466.776437] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1469.318300] Key type lgssc unregistered [ 1469.595849] LNet: 49149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.606196] LNetError: 49149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.615148] LNet: Removed LNI 192.168.204.148@tcp [ 1470.415301] Key type .llcrypt unregistered [ 1470.419299] Key type ._llcrypt unregistered [ 1492.553906] Key type ._llcrypt registered [ 1492.555895] Key type .llcrypt registered [ 1492.677914] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1493.711832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1493.888699] alg: No test for adler32 (adler32-zlib) [ 1494.912830] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1495.135522] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1496.815211] Key type lgssc registered [ 1497.759252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1504.708697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1504.736956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.209782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.922330] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1516.513292] LustreError: 50544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1516.524849] LustreError: 50544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1517.068915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1517.370939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1520.659190] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1525.993429] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1529.459308] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1535.211483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1535.643342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.878809] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1545.335190] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1549.560321] LustreError: 52782:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1549.671909] Lustre: server umount lustre-OST0000 complete [ 1553.376311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1553.384575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.395415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.962194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.966292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.802439] LustreError: 52983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1558.805383] LustreError: 52983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1558.896751] Lustre: server umount lustre-MDT0000 complete [ 1561.617945] LustreError: 50525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343321 with bad export cookie 8716742812001256004 [ 1561.625837] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.627329] LustreError: 50525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1570.795561] LustreError: 53185:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1570.803587] LustreError: 53185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1570.957979] Lustre: server umount lustre-MDT0001 complete [ 1580.505509] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1583.444238] Key type lgssc unregistered [ 1583.678432] LNet: 53720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.683709] LNetError: 53720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.697464] LNet: Removed LNI 192.168.204.148@tcp [ 1584.675244] Key type .llcrypt unregistered [ 1584.676900] Key type ._llcrypt unregistered [ 1606.537084] Key type ._llcrypt registered [ 1606.541432] Key type .llcrypt registered [ 1606.656262] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1607.409120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1607.521538] alg: No test for adler32 (adler32-zlib) [ 1608.659561] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1608.943277] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1610.631275] Key type lgssc registered [ 1611.642795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.578609] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.606933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1635.168836] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1640.287249] LustreError: 55086:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 1666.977498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1672.324246] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1677.604967] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1683.469867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.956619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.015115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1687.327633] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1689.058305] LustreError: 55113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1689.073551] LustreError: 55113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1694.179977] LustreError: 55113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.003724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.372620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.865243] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1703.757207] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1706.744284] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1709.536837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1709.549940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.562750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.560746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.581516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.602709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.097106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.615104] LustreError: 57361:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1714.732962] Lustre: server umount lustre-OST0000 complete [ 1736.160219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1736.167755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1736.195512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.198392] Lustre: Skipped 1 previous similar message [ 1741.236330] LustreError: 57736:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1741.243995] LustreError: 57736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1741.345854] Lustre: server umount lustre-MDT0000 complete [ 1742.311908] LustreError: 56009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.339458] LustreError: 56009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1744.593249] LustreError: 55093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343504 with bad export cookie 956700622745581964 [ 1744.615836] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.833779] LustreError: 57938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1750.847845] LustreError: 57938:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1751.061962] Lustre: server umount lustre-MDT0001 complete [ 1760.911853] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1763.455177] Key type lgssc unregistered [ 1763.757983] LNet: 58470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.770347] LNetError: 58470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.839762] LNet: Removed LNI 192.168.204.148@tcp [ 1765.539257] Key type .llcrypt unregistered [ 1765.544827] Key type ._llcrypt unregistered [ 1785.197576] Key type ._llcrypt registered [ 1785.199016] Key type .llcrypt registered [ 1785.286740] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1786.046731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1786.193076] alg: No test for adler32 (adler32-zlib) [ 1787.201511] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1787.418601] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1789.079168] Key type lgssc registered [ 1789.971238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.481884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1796.506117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.963972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.000552] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1806.890662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1807.255019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1809.720269] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1813.925775] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1817.138568] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1823.130257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1823.515616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.587865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1829.135136] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1834.300469] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.337734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1850.340605] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.350551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.414711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1853.419656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1853.420387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.064656] LustreError: 62114:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1856.194238] Lustre: server umount lustre-OST0000 complete [ 1863.651591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.652360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.652483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1863.660195] Lustre: Skipped 1 previous similar message [ 1863.662180] Lustre: Skipped 1 previous similar message [ 1868.769954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.774889] Lustre: Skipped 2 previous similar messages [ 1872.865341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.869501] Lustre: Skipped 1 previous similar message [ 1873.376180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1873.439631] LustreError: 62315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1873.445462] LustreError: 62315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1873.519457] Lustre: server umount lustre-MDT0000 complete [ 1876.277780] LustreError: 59827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343636 with bad export cookie 15988674745882789095 [ 1876.278574] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.295589] LustreError: 59827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1876.402655] LustreError: 62517:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1876.410082] LustreError: 62517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1876.513689] Lustre: server umount lustre-MDT0001 complete [ 1884.806972] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1886.676816] Key type lgssc unregistered [ 1886.919851] LNet: 63049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.926627] LNetError: 63049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.938669] LNet: Removed LNI 192.168.204.148@tcp [ 1887.538170] Key type .llcrypt unregistered [ 1887.546718] Key type ._llcrypt unregistered [ 1914.004695] Key type ._llcrypt registered [ 1914.006069] Key type .llcrypt registered [ 1914.081250] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1914.614763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.650833] alg: No test for adler32 (adler32-zlib) [ 1915.763414] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1915.963107] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1917.616605] Key type lgssc registered [ 1918.406593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.358038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.386237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.832740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1929.766487] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1936.691165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1937.128845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.645131] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1945.154425] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1948.449208] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1954.651603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.081369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.803138] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1964.415575] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1973.495265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1979.184774] Lustre: Failing over lustre-MDT0000 [ 1979.305434] LustreError: 66979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1979.405203] Lustre: server umount lustre-MDT0000 complete [ 1980.191892] LustreError: 64709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.214637] LustreError: 64709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1983.455991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1983.466643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.513460] LustreError: 64712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.982507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1985.279079] LustreError: 64708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.302127] LustreError: 64708:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1987.812828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.959082] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1988.113370] Lustre: *** cfs_fail_loc=135, val=0*** [ 1988.239128] LustreError: 67489:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1988.289911] LustreError: 67489:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -2 [ 1988.294045] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1988.294045] [ 1988.298217] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1988.321382] LustreError: 67477:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1988.336901] LustreError: 67477:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1988.349697] LustreError: 67477:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1988.381457] Lustre: server umount lustre-MDT0000 complete [ 1988.384647] LustreError: 67477:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1989.089891] LustreError: 64712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.107901] LustreError: 64712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1993.758411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.836643] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.055720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.107550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1995.521279] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1998.246111] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1999.348976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1999.376405] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1999.445124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2004.449020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.459585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.487634] Lustre: Skipped 2 previous similar messages [ 2006.398172] LustreError: 68297:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2006.402146] LustreError: 68297:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2006.494924] Lustre: server umount lustre-OST0000 complete [ 2014.688090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.694410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.699587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.708918] Lustre: Skipped 1 previous similar message [ 2014.718085] Lustre: Skipped 2 previous similar messages [ 2016.346636] LustreError: 68498:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2016.353042] LustreError: 68498:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2016.439357] Lustre: server umount lustre-MDT0000 complete [ 2019.241680] LustreError: 64694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343779 with bad export cookie 10467324279970789771 [ 2019.247390] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.247737] LustreError: 64694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2019.351804] LustreError: 68699:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2019.356691] LustreError: 68699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2019.487670] Lustre: server umount lustre-MDT0001 complete [ 2028.347644] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2031.119686] Key type lgssc unregistered [ 2031.428522] LNet: 69231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.438392] LNetError: 69231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.451126] LNet: Removed LNI 192.168.204.148@tcp [ 2032.378885] Key type .llcrypt unregistered [ 2032.382627] Key type ._llcrypt unregistered [ 2052.376612] Key type ._llcrypt registered [ 2052.378585] Key type .llcrypt registered [ 2052.459313] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2053.151821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.180138] alg: No test for adler32 (adler32-zlib) [ 2054.298211] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2054.525408] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2056.183169] Key type lgssc registered [ 2057.224380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2064.367176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2064.391585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.770781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.812581] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2071.011982] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.022305] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2075.114949] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.346280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2075.614208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.526649] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2082.930560] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2086.163068] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2091.789293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.111806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.223297] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2100.155923] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2110.574722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2114.154622] Lustre: Failing over lustre-MDT0000 [ 2114.288485] LustreError: 72868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2114.365634] Lustre: server umount lustre-MDT0000 complete [ 2115.866246] LustreError: 70598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2116.066416] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.083336] Lustre: Skipped 2 previous similar messages [ 2120.903386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.967164] LustreError: 70600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.980507] LustreError: 70600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2121.024573] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.117680] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2121.187669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2121.192928] Lustre: Skipped 2 previous similar messages [ 2121.211399] LustreError: 73378:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2121.215876] LustreError: 73378:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -9 [ 2121.220540] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2121.220540] [ 2121.224194] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2121.231402] LustreError: 73366:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2121.235870] LustreError: 73366:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2121.242095] LustreError: 73366:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2121.261377] Lustre: server umount lustre-MDT0000 complete [ 2121.263277] LustreError: 73366:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2125.730904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2125.818194] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.031717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.111265] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2129.074217] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2131.207281] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2131.436184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2131.457167] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2131.498353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2136.547325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.550689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2136.558490] Lustre: Skipped 1 previous similar message [ 2136.566255] Lustre: Skipped 1 previous similar message [ 2140.667234] LustreError: 74191:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2140.670545] LustreError: 74191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2140.751535] Lustre: server umount lustre-OST0000 complete [ 2146.785759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.793444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.798291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2146.801104] Lustre: Skipped 1 previous similar message [ 2149.472696] LustreError: 74394:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2149.477144] LustreError: 74394:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2149.546539] Lustre: server umount lustre-MDT0000 complete [ 2151.907477] LustreError: 70598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.921941] LustreError: 70598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2151.933036] LustreError: 73381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343912 with bad export cookie 12547697826350131886 [ 2151.937195] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.937798] LustreError: 73381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2152.061986] LustreError: 74596:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2152.065289] LustreError: 74596:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2152.190782] Lustre: server umount lustre-MDT0001 complete [ 2158.493216] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2160.600664] Key type lgssc unregistered [ 2160.826872] LNet: 75127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.833520] LNetError: 75127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.849590] LNet: Removed LNI 192.168.204.148@tcp [ 2161.449763] Key type .llcrypt unregistered [ 2161.451144] Key type ._llcrypt unregistered [ 2170.265408] Key type ._llcrypt registered [ 2170.267108] Key type .llcrypt registered [ 2170.355432] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2170.536652] Key type .llcrypt unregistered [ 2170.537910] Key type ._llcrypt unregistered [ 2186.387856] Key type ._llcrypt registered [ 2186.389239] Key type .llcrypt registered [ 2186.470322] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2187.252470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.338105] alg: No test for adler32 (adler32-zlib) [ 2188.353734] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2188.608266] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2190.288698] Key type lgssc registered [ 2191.117645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2196.823586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2196.848474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.260475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.941320] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2206.721475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.948866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2209.724054] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2213.595290] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2216.192378] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2221.780896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2222.193207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.215217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2226.864434] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2231.160736] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2238.946834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2238.949391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.969615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2243.043888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2243.045597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.048851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2244.730314] LustreError: 79228:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2244.815779] Lustre: server umount lustre-OST0000 complete [ 2248.163940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.164996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2248.182492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.204147] Lustre: Skipped 1 previous similar message [ 2253.280066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.283517] Lustre: Skipped 1 previous similar message [ 2253.719459] LustreError: 79429:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2253.724238] LustreError: 79429:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2253.779433] Lustre: server umount lustre-MDT0000 complete [ 2256.188707] LustreError: 76963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344016 with bad export cookie 9661996652049807358 [ 2256.193980] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.195433] LustreError: 76963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2256.304360] LustreError: 79630:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2256.307429] LustreError: 79630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2256.460771] Lustre: server umount lustre-MDT0001 complete [ 2267.465105] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2270.922834] Key type lgssc unregistered [ 2271.163913] LNet: 80162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.177750] LNetError: 80162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.197049] LNet: Removed LNI 192.168.204.148@tcp [ 2272.302415] Key type .llcrypt unregistered [ 2272.305343] Key type ._llcrypt unregistered [ 2296.371244] Key type ._llcrypt registered [ 2296.374347] Key type .llcrypt registered [ 2296.462664] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2298.129742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2298.145678] alg: No test for adler32 (adler32-zlib) [ 2299.174625] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2299.472058] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2301.119211] Key type lgssc registered [ 2302.096820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2309.921689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2309.943879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.495312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.747539] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2320.869944] LustreError: 81562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.883379] LustreError: 81562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2323.336170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2323.654222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2327.192279] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2331.194262] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2333.579198] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.429688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.669078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.725227] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2346.415173] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2356.089831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2363.363142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2363.370855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2363.407953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.436649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2366.439801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.442295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2366.452336] Lustre: Skipped 1 previous similar message [ 2368.761151] LustreError: 83828:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2368.922912] Lustre: server umount lustre-OST0000 complete [ 2374.623899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2374.636804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.648425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.676886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.683175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.714540] Lustre: Skipped 1 previous similar message [ 2378.525222] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2378.529606] LustreError: 84030:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2378.585699] Lustre: server umount lustre-MDT0000 complete [ 2381.401514] LustreError: 81543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344141 with bad export cookie 14672891121840052770 [ 2381.409214] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2381.416202] LustreError: 81543:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2381.576286] LustreError: 84233:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2381.579393] LustreError: 84233:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2381.757810] Lustre: server umount lustre-MDT0001 complete [ 2391.021094] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2393.262816] Key type lgssc unregistered [ 2393.520277] LNet: 84766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.525678] LNetError: 84766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.539633] LNet: Removed LNI 192.168.204.148@tcp [ 2394.260660] Key type .llcrypt unregistered [ 2394.262242] Key type ._llcrypt unregistered [ 2412.970280] Key type ._llcrypt registered [ 2412.971634] Key type .llcrypt registered [ 2413.046703] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2413.544085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2413.611557] alg: No test for adler32 (adler32-zlib) [ 2414.573184] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2414.765955] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2416.415159] Key type lgssc registered [ 2417.357456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2423.829762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2423.851151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.264529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.323957] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2434.419417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2434.786802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2437.917641] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2441.819725] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2444.932785] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2450.359818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2450.648153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.689822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2454.855981] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2459.200968] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2469.345262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.347196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.348906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.376832] Lustre: Skipped 1 previous similar message [ 2470.083181] LustreError: 88385:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2470.158770] Lustre: server umount lustre-OST0000 complete [ 2474.465871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.485914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.489068] Lustre: Skipped 1 previous similar message [ 2479.259384] LustreError: 88585:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2479.262083] LustreError: 88585:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2479.316523] Lustre: server umount lustre-MDT0000 complete [ 2479.585317] LustreError: 86137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.599857] LustreError: 86137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2481.870616] LustreError: 86119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344242 with bad export cookie 10659833713346074722 [ 2481.874623] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2481.876129] LustreError: 86119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2481.967169] LustreError: 88787:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2481.972392] LustreError: 88787:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2482.117489] Lustre: server umount lustre-MDT0001 complete [ 2490.682512] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2494.053017] Key type lgssc unregistered [ 2494.563278] LNet: 89320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.579037] LNetError: 89320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2494.616688] LNet: Removed LNI 192.168.204.148@tcp [ 2495.489216] Key type .llcrypt unregistered [ 2495.496774] Key type ._llcrypt unregistered [ 2516.964217] Key type ._llcrypt registered [ 2516.966534] Key type .llcrypt registered [ 2517.039678] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2517.856230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2517.893302] alg: No test for adler32 (adler32-zlib) [ 2519.190264] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2519.423940] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2521.146443] Key type lgssc registered [ 2522.037763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2528.965915] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2528.979919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2544.479412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2549.599214] LustreError: 90696:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 2576.352205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2580.936691] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2585.430817] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.329567] LustreError: 91553:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2589.416925] Lustre: server umount lustre-OST0000 complete [ 2595.560472] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:52:35 (1763344355) [ 2601.585472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.124886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.909236] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2607.585361] LustreError: 92223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.597973] LustreError: 92223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2611.930810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2612.135861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.413765] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2617.846220] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2620.202678] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.525156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2624.748722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2627.830560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2628.469571] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2632.546610] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2647.523160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.526539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2647.531237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2647.532221] Lustre: Skipped 1 previous similar message [ 2650.744131] LustreError: 94629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2650.747226] LustreError: 94629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2650.872234] Lustre: server umount lustre-OST0000 complete [ 2657.763513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.767618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.768490] Lustre: Skipped 1 previous similar message [ 2657.772683] Lustre: Skipped 2 previous similar messages [ 2660.033899] LustreError: 94832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2660.042319] LustreError: 94832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2660.203286] Lustre: server umount lustre-MDT0000 complete [ 2662.880574] LustreError: 92223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.902737] LustreError: 92223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2664.312491] LustreError: 92943:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344424 with bad export cookie 13841501756672582105 [ 2664.320723] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2664.335761] LustreError: 92943:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2664.502670] LustreError: 95033:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2664.505068] LustreError: 95033:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2664.647695] Lustre: server umount lustre-MDT0001 complete [ 2672.819065] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2675.564866] Key type lgssc unregistered [ 2675.880581] LNet: 95567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.895452] LNetError: 95567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.924675] LNet: Removed LNI 192.168.204.148@tcp [ 2676.870413] Key type .llcrypt unregistered [ 2676.872128] Key type ._llcrypt unregistered [ 2698.217825] Key type ._llcrypt registered [ 2698.219260] Key type .llcrypt registered [ 2698.360609] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2699.270064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2699.364195] alg: No test for adler32 (adler32-zlib) [ 2700.507309] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2700.755317] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2702.497862] Key type lgssc registered [ 2703.714563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2710.876849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2710.899462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.307178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.527540] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2717.673172] LustreError: 96957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.684605] LustreError: 96957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2721.766684] LustreError: 96957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.204487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2722.492932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2725.491365] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2729.704481] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.416846] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2737.472605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2737.707941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.391650] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2746.540689] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2757.165977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2762.363098] LustreError: 99226:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2762.481029] Lustre: server umount lustre-OST0000 complete [ 2767.329748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.331438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.349281] Lustre: Skipped 1 previous similar message [ 2767.359112] Lustre: Skipped 1 previous similar message [ 2772.449337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.451644] Lustre: Skipped 1 previous similar message [ 2777.568721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.578896] Lustre: Skipped 1 previous similar message [ 2780.127118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2780.236749] LustreError: 99426:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2780.244845] LustreError: 99426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2780.344620] Lustre: server umount lustre-MDT0000 complete [ 2782.692776] LustreError: 97668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.704405] LustreError: 97668:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2783.000719] LustreError: 96938:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344543 with bad export cookie 13616395979103470747 [ 2783.008626] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2793.451981] LustreError: 99630:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2793.455526] LustreError: 99630:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2793.578151] Lustre: server umount lustre-MDT0001 complete [ 2800.391776] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2802.309190] Key type lgssc unregistered [ 2802.532993] LNet: 100162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2802.538792] LNetError: 100162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2802.552385] LNet: Removed LNI 192.168.204.148@tcp [ 2803.081312] Key type .llcrypt unregistered [ 2803.083576] Key type ._llcrypt unregistered [ 2816.939347] Key type ._llcrypt registered [ 2816.949431] Key type .llcrypt registered [ 2817.045964] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2817.693856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2817.704274] alg: No test for adler32 (adler32-zlib) [ 2818.665323] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 2818.870482] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2820.527145] Key type lgssc registered [ 2821.359606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2827.154912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2827.166068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2842.655458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2847.779603] LustreError: 101365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 2874.463529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.945862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2884.027621] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2889.838696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.173851] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2890.186251] LustreError: 102280:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2890.191395] LustreError: 102280:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2890.202746] LustreError: 102280:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2890.303502] Lustre: server umount lustre-MDT0000 complete [ 2890.306863] LustreError: 102280:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2911.711187] Lustre: 100704:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344655/real 1763344655] req@0000000067096d71 x1849000805272832/t0(0) o400->MGC192.168.204.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763344671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2911.711356] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2911.755757] Lustre: 100704:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2911.760389] LustreError: 101379:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2932.192453] LustreError: 102710:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2932.364828] Lustre: server umount lustre-OST0000 complete [ 2940.788185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2947.712495] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2955.711844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2960.902312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2965.927343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.488728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2978.024958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2978.109993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.335845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2978.375317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2978.488691] Lustre: lustre-MDT0000: new disk, initializing [ 2978.587764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.604699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2982.384641] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2992.029869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.106321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.195938] Lustre: 105053:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2992.277393] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2992.293826] Lustre: Skipped 1 previous similar message [ 2992.447558] Lustre: lustre-MDT0001: new disk, initializing [ 2992.551609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.584690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2992.589470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2995.635093] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3001.525276] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3004.318092] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3008.289198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.356637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3008.536159] Lustre: lustre-OST0000: new disk, initializing [ 3008.538774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3008.588374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.493119] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3017.804432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3017.818119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3017.960300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3018.473297] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.695638] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3021.922686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3025.036364] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3025.240788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3028.452495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.454677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.483605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.571506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.574447] Lustre: Skipped 2 previous similar messages [ 3038.690353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3041.247157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.277912] LustreError: 107993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3041.283161] LustreError: 107993:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3041.361778] Lustre: server umount lustre-OST0000 complete [ 3048.928369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.934891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.937490] Lustre: Skipped 2 previous similar messages [ 3048.946684] Lustre: Skipped 1 previous similar message [ 3050.664941] LustreError: 108195:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3050.682464] LustreError: 108195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3050.797130] Lustre: server umount lustre-MDT0000 complete [ 3054.462685] LustreError: 105045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344814 with bad export cookie 9221468612927319554 [ 3054.469507] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.482148] LustreError: 105045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3054.665020] LustreError: 108398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3054.670687] LustreError: 108398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3054.838346] Lustre: server umount lustre-MDT0001 complete [ 3061.800597] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:00:21 (1763344821) [ 3064.910178] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3079.803759] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3087.257337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3095.250750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3100.889932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3106.966466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.515910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3118.856720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3118.926369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.300099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3119.367102] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3119.499456] Lustre: lustre-MDT0000: new disk, initializing [ 3119.580862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.594323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3123.301646] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3132.497511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.586433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3132.666120] Lustre: 111547:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3132.726987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3132.729639] Lustre: Skipped 1 previous similar message [ 3132.828904] Lustre: lustre-MDT0001: new disk, initializing [ 3132.913044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.963018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3132.969591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3136.274721] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3142.946355] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.994583] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3151.713859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3151.775522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3152.005894] Lustre: lustre-OST0000: new disk, initializing [ 3152.008413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3152.091434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.179603] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3161.160384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3161.163929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3161.260629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3161.550065] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3164.935881] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3165.136549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3168.040614] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3168.204850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3171.810542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3171.819707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.820677] LustreError: Skipped 1 previous similar message [ 3171.823600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3171.837142] Lustre: Skipped 1 previous similar message [ 3171.845667] Lustre: Skipped 1 previous similar message [ 3175.796446] LustreError: 114479:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3175.802675] LustreError: 114479:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3175.950558] Lustre: server umount lustre-OST0000 complete [ 3178.977184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3178.987530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3182.048295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3182.056072] Lustre: Skipped 3 previous similar messages [ 3193.311658] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3193.377564] LustreError: 114682:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3193.379955] LustreError: 114682:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3193.449572] Lustre: server umount lustre-MDT0000 complete [ 3196.752287] LustreError: 111539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344957 with bad export cookie 9221468612927320436 [ 3196.758726] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3196.764968] LustreError: 111539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3197.110912] Lustre: server umount lustre-MDT0001 complete [ 3204.010111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.326064] LustreError: 115363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.338060] LustreError: 115363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3204.393942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.459939] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3209.698147] LustreError: 115362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.799829] LustreError: 115362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.185382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3218.039851] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3222.043546] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.851123] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3229.370899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3234.296378] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3239.456683] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3255.322637] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3259.367812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.368957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3259.369013] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.384976] Lustre: Skipped 2 previous similar messages [ 3259.418534] Lustre: Skipped 5 previous similar messages [ 3263.895099] LustreError: 117725:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3263.905797] LustreError: 117725:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3264.062717] Lustre: server umount lustre-OST0000 complete [ 3269.603103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.612943] Lustre: Skipped 1 previous similar message [ 3282.399116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.557646] Lustre: server umount lustre-MDT0000 complete [ 3283.941472] LustreError: 115363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.959956] LustreError: 115363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3285.389803] LustreError: 115342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345045 with bad export cookie 9221468612927321318 [ 3285.397732] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.408969] LustreError: 115342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3285.669834] Lustre: server umount lustre-MDT0001 complete [ 3294.506135] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3296.363500] Key type lgssc unregistered [ 3296.561985] LNet: 118664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.566576] LNetError: 118664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.576769] LNet: Removed LNI 192.168.204.148@tcp [ 3297.154667] Key type .llcrypt unregistered [ 3297.156688] Key type ._llcrypt unregistered [ 3310.054267] Key type ._llcrypt registered [ 3310.055770] Key type .llcrypt registered [ 3310.141873] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3321.633936] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3322.099099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3322.111864] alg: No test for adler32 (adler32-zlib) [ 3323.115277] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 3323.323744] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3324.991162] Key type lgssc registered [ 3325.846667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3333.491979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3338.656833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3343.547107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3347.917957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.911723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3352.981844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3352.992708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.223623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3354.271616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3354.358626] Lustre: lustre-MDT0000: new disk, initializing [ 3354.430948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.453895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3358.050895] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3367.158893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3367.226150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3367.283702] Lustre: 121770:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3367.337947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3367.340823] Lustre: Skipped 1 previous similar message [ 3367.406925] Lustre: lustre-MDT0001: new disk, initializing [ 3367.491570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.557459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3367.578582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3370.917833] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3378.209240] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.721321] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3387.677572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3387.722451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3387.913366] Lustre: lustre-OST0000: new disk, initializing [ 3387.924384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3388.002421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.532993] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3396.631111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3396.642728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3396.748335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3398.160802] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3401.441141] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3401.613523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3404.868697] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3405.043497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3406.825192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.828460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.831451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.939934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.952626] Lustre: Skipped 1 previous similar message [ 3412.543446] LustreError: 124706:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3412.645983] Lustre: server umount lustre-OST0000 complete [ 3417.057990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.075512] Lustre: Skipped 1 previous similar message [ 3417.084812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3417.090322] Lustre: Skipped 2 previous similar messages [ 3421.390182] LustreError: 124907:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3421.394488] LustreError: 124907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3421.480933] Lustre: server umount lustre-MDT0000 complete [ 3424.101024] LustreError: 121760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345184 with bad export cookie 12886547580221004 [ 3424.106478] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.109305] LustreError: 121760:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3424.234081] LustreError: 125108:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3424.238191] LustreError: 125108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3424.360594] Lustre: server umount lustre-MDT0001 complete [ 3430.545785] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:06:30 (1763345190) [ 3436.149039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.539992] LustreError: 125781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.553844] LustreError: 125781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3436.594971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.670215] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3441.632560] LustreError: 125781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.258295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3446.551399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3449.473470] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3453.540501] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3456.020292] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3457.553285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.567699] Lustre: Skipped 1 previous similar message [ 3457.579022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.585110] Lustre: Skipped 1 previous similar message [ 3462.628670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.638421] Lustre: Skipped 1 previous similar message [ 3463.714195] LustreError: 127122:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3463.717467] LustreError: 127122:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3463.824042] Lustre: server umount lustre-MDT0000 complete [ 3466.789873] LustreError: 125762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345227 with bad export cookie 12886547580221886 [ 3466.800661] LustreError: 125762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3466.805601] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.067171] Lustre: server umount lustre-MDT0001 complete [ 3473.744391] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:07:13 (1763345233) [ 3479.267071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3494.816243] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3499.935457] LustreError: 127967:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 3525.599868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.683610] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3533.655520] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3535.250092] LustreError: 128633:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3535.259071] LustreError: 128633:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3535.388545] Lustre: server umount lustre-OST0000 complete [ 3541.768616] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:08:21 (1763345301) [ 3547.091679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.388781] LustreError: 129303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.448664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.952993] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3552.736617] LustreError: 129303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3554.977268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3557.710728] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3561.568403] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3564.544717] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3570.174314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3575.528738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3580.257981] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3586.016860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.024488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3586.055286] Lustre: Skipped 3 previous similar messages [ 3586.058027] Lustre: Skipped 1 previous similar message [ 3592.090309] LustreError: 131548:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3592.094921] LustreError: 131548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3592.201055] Lustre: server umount lustre-MDT0000 complete [ 3595.554640] LustreError: 129283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345355 with bad export cookie 12886547580222551 [ 3595.561516] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3595.570770] LustreError: 129283:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3595.893470] Lustre: server umount lustre-MDT0001 complete [ 3609.556402] LustreError: 131950:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3609.558984] LustreError: 131950:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3609.729773] Lustre: server umount lustre-OST0000 complete [ 3616.849341] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:09:36 (1763345376) [ 3623.013783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.361198] LustreError: 132622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3623.436451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.439417] Lustre: Skipped 2 previous similar messages [ 3626.971724] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3633.635414] LustreError: 132621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.650707] LustreError: 132621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3633.833179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3637.198746] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3641.507317] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3644.445802] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.124792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3655.229903] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3659.516639] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3662.755234] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3667.889032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3668.206441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3670.803674] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3670.959760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3673.058464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.059555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.070728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.074371] Lustre: Skipped 3 previous similar messages [ 3674.305712] LustreError: 135399:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3674.310450] LustreError: 135399:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3674.372285] Lustre: server umount lustre-OST0000 complete [ 3683.467896] Lustre: server umount lustre-MDT0000 complete [ 3686.775747] LustreError: 132602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345447 with bad export cookie 12886547580223622 [ 3686.781101] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.800211] LustreError: 132602:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3687.107171] Lustre: server umount lustre-MDT0001 complete [ 3694.045391] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:10:53 (1763345453) [ 3699.268350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3714.789495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3719.903652] LustreError: 136448:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 3745.567978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3745.574318] Lustre: Skipped 2 previous similar messages [ 3750.657495] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3755.877584] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3761.735957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.081345] LustreError: 136475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3762.276461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3765.480402] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3772.199057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3772.393031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3772.398943] Lustre: Skipped 4 previous similar messages [ 3776.084793] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3780.886658] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3784.087411] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3787.963530] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3788.149243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3791.392839] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3791.539310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3795.425158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.449750] Lustre: Skipped 5 previous similar messages [ 3799.154353] LustreError: 139222:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3799.156469] LustreError: 139222:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3799.316489] Lustre: server umount lustre-MDT0000 complete [ 3800.549072] LustreError: 138492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3800.571468] LustreError: 138492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3802.241284] LustreError: 136455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345562 with bad export cookie 12886547580224644 [ 3802.245703] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3802.250024] LustreError: 136455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3802.520623] Lustre: server umount lustre-MDT0001 complete [ 3815.506516] Lustre: server umount lustre-OST0000 complete [ 3822.960433] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:13:02 (1763345582) [ 3827.728665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3843.231569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3848.351479] LustreError: 140271:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 3874.079263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.081904] Lustre: Skipped 2 previous similar messages [ 3879.422397] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3884.650224] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3890.474326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.926072] LustreError: 140297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.115949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3895.273775] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3903.025693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3906.821344] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3911.076868] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3913.976535] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3919.651091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3919.751476] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3919.932364] Lustre: lustre-OST0001: new disk, initializing [ 3919.941698] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3925.000798] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3927.640919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3927.678903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3927.708525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3931.848734] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3935.980755] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3936.218801] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3940.501297] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3940.696950] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3942.886298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.921116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.944775] Lustre: Skipped 3 previous similar messages [ 3947.489501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.217241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3957.275346] LustreError: 144005:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3957.278784] LustreError: 144005:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3957.331890] Lustre: server umount lustre-OST0000 complete [ 3966.490432] Lustre: server umount lustre-OST0001 complete [ 3970.016638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3984.351140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3984.531893] Lustre: server umount lustre-MDT0000 complete [ 3987.485332] LustreError: 140278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345747 with bad export cookie 12886547580225589 [ 3987.490889] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.493884] LustreError: 140278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3987.735574] Lustre: server umount lustre-MDT0001 complete [ 3992.422819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.851081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3998.869038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4001.646919] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.990182] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:16:06 (1763345766) [ 4008.714228] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4010.086953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4011.311512] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:16:11 (1763345771) [ 4017.046244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.244538] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4017.266776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4021.185266] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4029.212046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4029.299811] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4029.318903] Lustre: 146252:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4033.359552] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4038.164232] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4041.360529] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4047.367825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4047.595504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4053.182389] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4054.722576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4058.528607] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.537964] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4062.800697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4066.264560] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4066.519769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4069.858484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.871118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.878166] Lustre: Skipped 5 previous similar messages [ 4074.332402] Lustre: server umount lustre-OST0000 complete [ 4094.554418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4100.070870] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4100.080952] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4100.136828] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4104.446566] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4108.385833] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4110.323706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.335629] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4110.619130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4114.974654] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4115.173388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4118.627262] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c7a488bf800.ost_server_uuid 50 [ 4120.115498] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c7a488bf800.ost_server_uuid in FULL state after 0 sec [ 4120.311913] LustreError: 146258:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.48@tcp) returned error from blocking AST (req@000000002a090e50 x1849001333982976 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000bc8b5781/0x2dc83f2ae69188 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.48@tcp remote: 0xf36d0bb82712c46e expref: 5 pid: 146258 timeout: 4220 lvb_type: 0 [ 4120.330547] LustreError: lustre-MDT0000: A client on nid 192.168.204.48@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4120.336279] LustreError: 146247:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.48@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000bc8b5781/0x2dc83f2ae69188 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.48@tcp remote: 0xf36d0bb82712c46e expref: 6 pid: 146258 timeout: 0 lvb_type: 0 [ 4125.665328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4125.678096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4125.679776] LustreError: Skipped 3 previous similar messages [ 4125.682020] Lustre: Skipped 17 previous similar messages [ 4129.583804] Lustre: server umount lustre-OST0000 complete [ 4135.909226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.919095] Lustre: Skipped 4 previous similar messages [ 4138.365163] Lustre: server umount lustre-MDT0000 complete [ 4141.036768] LustreError: 146257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.047516] LustreError: 146257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4141.232909] LustreError: 146243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345901 with bad export cookie 12886547580226765 [ 4141.238514] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.257082] LustreError: 146243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4147.658946] Lustre: server umount lustre-MDT0001 complete [ 4154.782088] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4157.112934] Key type lgssc unregistered [ 4157.358640] LNet: 151750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4157.367201] LNetError: 151750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4157.384822] LNet: Removed LNI 192.168.204.148@tcp [ 4158.086444] Key type .llcrypt unregistered [ 4158.088489] Key type ._llcrypt unregistered [ 4177.963598] Key type ._llcrypt registered [ 4177.964793] Key type .llcrypt registered [ 4178.099107] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4179.014179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4179.094495] alg: No test for adler32 (adler32-zlib) [ 4180.163962] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4180.328404] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4181.999174] Key type lgssc registered [ 4182.983866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4189.501086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4189.528555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4190.941711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.904850] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4200.126930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4200.416153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4203.407622] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4206.870701] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4209.454638] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4214.389030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4214.669767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4216.752091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4218.404412] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4222.300323] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4231.863932] Lustre: Failing over lustre-MDT0000 [ 4232.050478] LustreError: 155370:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4232.215621] Lustre: server umount lustre-MDT0000 complete [ 4235.235199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4235.240202] LustreError: 153120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4235.246803] LustreError: 153120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4240.356268] LustreError: 153837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.373289] LustreError: 153837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4244.312124] LustreError: 153103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.48@tcp arrived at 1763346004 with bad export cookie 15691203084440327449 [ 4244.336734] LustreError: 153103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4245.472256] LustreError: 153837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4245.482764] LustreError: 153837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4249.908539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.058984] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4250.298210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4250.383847] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4253.440561] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4255.145249] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4255.153047] Lustre: lustre-MDT0000: Denying connection for new client 1ab25ada-a2c0-4626-ab62-e052e0ffccff (at 192.168.204.48@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4255.724357] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4265.955102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4265.958822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4265.960531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4265.968722] Lustre: Skipped 3 previous similar messages [ 4271.074843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4271.088151] Lustre: Skipped 2 previous similar messages [ 4276.196851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.204537] Lustre: Skipped 1 previous similar message [ 4277.215184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4277.259101] LustreError: 156307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4277.311482] LustreError: 156307:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4277.457725] Lustre: server umount lustre-OST0000 complete [ 4280.607213] LustreError: 156510:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4280.612113] Lustre: 155874:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4280.618269] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4280.655805] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4280.676521] LustreError: 155874:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c1553711 x1849002232422144/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4280.710397] LustreError: 155874:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4280.721611] LustreError: 155874:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4280.731232] Lustre: lustre-MDT0000: Recovery over after 0:25, of 2 clients 0 recovered and 2 were evicted. [ 4280.931107] LustreError: 156510:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4280.935919] LustreError: 156510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4281.100389] Lustre: server umount lustre-MDT0000 complete [ 4281.312829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.325490] LustreError: 153120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4284.118772] LustreError: 153104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346044 with bad export cookie 15691203084440327554 [ 4284.125454] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.135904] LustreError: 152432:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4284.137127] LustreError: 153104:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4286.314793] LustreError: 156711:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4286.340659] LustreError: 156711:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4286.520640] Lustre: server umount lustre-MDT0001 complete [ 4294.948325] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4297.306611] Key type lgssc unregistered [ 4297.615280] LNet: 157243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4297.623435] LNetError: 157243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4297.638897] LNet: Removed LNI 192.168.204.148@tcp [ 4298.406786] Key type .llcrypt unregistered [ 4298.409064] Key type ._llcrypt unregistered [ 4317.374867] Key type ._llcrypt registered [ 4317.377557] Key type .llcrypt registered [ 4317.482824] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4318.062655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4318.082584] alg: No test for adler32 (adler32-zlib) [ 4319.185390] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4319.415516] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4321.081952] Key type lgssc registered [ 4322.086560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4329.517735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4329.540939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.028475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.268105] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4341.178912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4341.218419] LustreError: 158634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.225224] LustreError: 158634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4341.455490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4344.935591] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4349.271797] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4352.210102] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4357.714094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4358.066245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.038129] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4368.087131] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.499239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4386.784590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4386.787708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.803284] Lustre: Skipped 1 previous similar message [ 4386.807993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.810568] Lustre: Skipped 1 previous similar message [ 4390.077859] LustreError: 160902:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4390.215439] Lustre: server umount lustre-OST0000 complete [ 4397.024301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.039157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.740790] LustreError: 161103:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4399.807122] LustreError: 161103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4399.920890] Lustre: server umount lustre-MDT0000 complete [ 4402.151691] LustreError: 159861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.171158] LustreError: 159861:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4403.476273] LustreError: 158614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346163 with bad export cookie 9182179691924589644 [ 4403.482414] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4403.486779] LustreError: 158614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4403.658721] LustreError: 161305:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4403.666559] LustreError: 161305:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4403.838362] Lustre: server umount lustre-MDT0001 complete [ 4412.659125] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4415.513087] Key type lgssc unregistered [ 4415.878773] LNet: 161836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4415.897539] LNetError: 161836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4416.935582] LNet: Removed LNI 192.168.204.148@tcp [ 4417.723212] Key type .llcrypt unregistered [ 4417.724782] Key type ._llcrypt unregistered [ 4447.264106] Key type ._llcrypt registered [ 4447.275575] Key type .llcrypt registered [ 4447.439605] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4448.771500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4448.793078] alg: No test for adler32 (adler32-zlib) [ 4449.923054] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4450.091799] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4451.865914] Key type lgssc registered [ 4453.069626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4461.534116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4461.572290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.138807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.729221] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4473.316599] LustreError: 163500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.336899] LustreError: 163500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4474.626912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4475.072734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.878288] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4483.046562] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4486.197322] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4492.660515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4493.205777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.353410] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4505.294572] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4508.622226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4514.491595] LustreError: 165747:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4514.586459] Lustre: server umount lustre-OST0000 complete [ 4518.882477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.888179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.902924] Lustre: Skipped 1 previous similar message [ 4518.920430] Lustre: Skipped 1 previous similar message [ 4524.001075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4524.009309] Lustre: Skipped 1 previous similar message [ 4528.112134] LustreError: 165949:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4528.119530] LustreError: 165949:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4528.201137] Lustre: server umount lustre-MDT0000 complete [ 4529.121491] LustreError: 163497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.133850] LustreError: 163497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4531.246783] LustreError: 164221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346291 with bad export cookie 12967438094854951317 [ 4531.251465] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.253807] LustreError: 164221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4541.427733] LustreError: 166152:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4541.456638] LustreError: 166152:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4541.633954] Lustre: server umount lustre-MDT0001 complete [ 4550.651580] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4553.355617] Key type lgssc unregistered [ 4553.652806] LNet: 166685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4553.659513] LNetError: 166685:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4554.734187] LNet: Removed LNI 192.168.204.148@tcp [ 4555.389265] Key type .llcrypt unregistered [ 4555.395710] Key type ._llcrypt unregistered [ 4572.841938] Key type ._llcrypt registered [ 4572.843108] Key type .llcrypt registered [ 4572.917351] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4573.735340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4573.756831] alg: No test for adler32 (adler32-zlib) [ 4574.870195] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4575.105975] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4576.759340] Key type lgssc registered [ 4577.950322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4587.040587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4587.068675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.634838] Lustre: *** cfs_fail_loc=135, val=0*** [ 4588.746832] LustreError: 168169:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4588.765663] LustreError: 168169:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.148@tcp: cfg command failed: rc = -2 [ 4588.773228] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4588.773228] [ 4588.785978] LustreError: MGC192.168.204.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4588.847064] LustreError: 168128:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4588.859438] LustreError: 168128:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4588.867575] LustreError: 168128:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4588.870843] LustreError: 168128:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.204.148@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000b35a993a) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4588.883526] LustreError: 168128:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4589.010933] Lustre: server umount lustre-MDT0000 complete [ 4589.012994] LustreError: 168128:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4595.188567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.796207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.238552] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4600.803245] LustreError: 168543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.828408] LustreError: 168543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4605.927319] LustreError: 168543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.682416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4608.193088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4611.826881] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4616.725544] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4619.694623] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4623.839935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4623.849848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.863861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.890759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.893898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.753502] LustreError: 169886:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4627.774986] LustreError: 169886:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4627.943946] Lustre: server umount lustre-MDT0000 complete [ 4631.011980] LustreError: 168538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.065883] LustreError: 168538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4631.834720] LustreError: 168524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346392 with bad export cookie 410935812799311745 [ 4631.843651] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4631.852666] LustreError: 168524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4632.056397] LustreError: 170089:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4632.151689] LustreError: 170089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4632.563061] Lustre: server umount lustre-MDT0001 complete [ 4643.438792] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4646.229985] Key type lgssc unregistered [ 4646.509232] LNet: 170622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4646.516762] LNetError: 170622:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4646.529657] LNet: Removed LNI 192.168.204.148@tcp [ 4647.149182] Key type .llcrypt unregistered [ 4647.150841] Key type ._llcrypt unregistered [ 4666.128170] Key type ._llcrypt registered [ 4666.129257] Key type .llcrypt registered [ 4666.297920] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4666.654480] Key type .llcrypt unregistered [ 4666.656272] Key type ._llcrypt unregistered [ 4681.854276] Key type ._llcrypt registered [ 4681.856862] Key type .llcrypt registered [ 4681.961821] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4682.887328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.917674] alg: No test for adler32 (adler32-zlib) [ 4684.068482] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4684.262180] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4685.904138] Key type lgssc registered [ 4686.933549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4694.024983] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4694.052379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4709.599278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4714.719205] LustreError: 172711:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 4741.407187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.751291] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4751.106918] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.753054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4758.293279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.354840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4762.075567] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4763.641746] LustreError: 172736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.660771] LustreError: 172736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4768.744378] LustreError: 172736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4769.777580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4770.148369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4774.379412] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4779.887745] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4782.791369] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4785.155578] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4799.457872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4799.462567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.474857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.993106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4801.008593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.020141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.021951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.025144] Lustre: Skipped 1 previous similar message [ 4803.128761] LustreError: 175433:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4803.243607] Lustre: server umount lustre-OST0000 complete [ 4807.136952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.140272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4807.154051] Lustre: Skipped 1 previous similar message [ 4812.258121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4812.266032] Lustre: Skipped 2 previous similar messages [ 4813.036578] LustreError: 175634:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4813.046216] LustreError: 175634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4813.165386] Lustre: server umount lustre-MDT0000 complete [ 4816.353407] LustreError: 175036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346576 with bad export cookie 4427713116415586713 [ 4816.356766] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4816.368475] LustreError: 175036:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4816.506778] LustreError: 175835:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4816.512112] LustreError: 175835:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4816.681886] Lustre: server umount lustre-MDT0001 complete [ 4827.092416] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4829.996138] Key type lgssc unregistered [ 4830.245563] LNet: 176367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4830.257029] LNetError: 176367:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4830.270744] LNet: Removed LNI 192.168.204.148@tcp [ 4830.930665] Key type .llcrypt unregistered [ 4830.932490] Key type ._llcrypt unregistered [ 4851.625843] Key type ._llcrypt registered [ 4851.627548] Key type .llcrypt registered [ 4851.785805] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4852.858831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4853.087938] alg: No test for adler32 (adler32-zlib) [ 4854.212628] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 4854.451253] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4856.151158] Key type lgssc registered [ 4857.271142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4863.474897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4863.495550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.919384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.039578] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4874.943527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4875.208760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.669575] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4884.537562] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4888.343811] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4894.586100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4895.004141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.725368] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4906.191908] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.432657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4916.129843] Lustre: Failing over lustre-MDT0000 [ 4916.192076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4916.196119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.215986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.390492] LustreError: 180069:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4916.528111] Lustre: server umount lustre-MDT0000 complete [ 4916.709957] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.762860] LustreError: 179675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.770238] LustreError: 179675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4918.029739] LustreError: 179675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.826871] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.842607] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4926.951714] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.975605] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4932.070258] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.086188] LustreError: 177772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4933.413451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4933.504807] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.675614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.719330] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4936.110682] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4938.497489] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4938.736150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4938.755235] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4938.816888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4939.359461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4944.009588] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4954.083417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.091301] Lustre: Skipped 1 previous similar message [ 4954.094395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4954.099982] Lustre: Skipped 1 previous similar message [ 4959.205245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4959.211222] Lustre: Skipped 1 previous similar message [ 4959.550605] LustreError: 181304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4959.552884] LustreError: 181304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4959.692083] Lustre: server umount lustre-OST0000 complete [ 4964.324798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4964.327151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.336089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.352698] Lustre: Skipped 2 previous similar messages [ 4969.263293] LustreError: 181506:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4969.272269] LustreError: 181506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4969.345933] Lustre: server umount lustre-MDT0000 complete [ 4969.444969] LustreError: 177766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.453035] LustreError: 177766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4972.619301] LustreError: 177753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346732 with bad export cookie 13490712278005007453 [ 4972.623303] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4972.627040] LustreError: 177753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4972.772107] LustreError: 181708:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4972.779546] LustreError: 181708:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4972.949345] Lustre: server umount lustre-MDT0001 complete [ 4982.566047] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4985.171373] Key type lgssc unregistered [ 4985.450110] LNet: 182241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4985.458699] LNetError: 182241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4985.482262] LNet: Removed LNI 192.168.204.148@tcp [ 4986.249606] Key type .llcrypt unregistered [ 4986.252953] Key type ._llcrypt unregistered [ 5006.122212] Key type ._llcrypt registered [ 5006.130947] Key type .llcrypt registered [ 5006.274711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5007.475324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5007.508049] alg: No test for adler32 (adler32-zlib) [ 5008.572536] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5008.882519] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5010.591873] Key type lgssc registered [ 5011.791842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5018.276942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5018.293603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.686306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.937966] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5028.841102] LustreError: 183616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5028.846621] LustreError: 183616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5030.548212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5030.902412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.966659] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5038.148946] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.189786] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5047.020611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5047.405602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.525793] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5057.240949] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5063.801563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5065.038662] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5069.305825] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5076.555846] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5087.718866] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.721394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.733458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.763273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.763799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.774605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5089.787162] Lustre: Skipped 1 previous similar message [ 5093.633673] LustreError: 186029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5093.730443] Lustre: server umount lustre-OST0000 complete [ 5097.441218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5097.447680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.473187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.001708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5100.011157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5105.121176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.129159] Lustre: Skipped 2 previous similar messages [ 5111.264310] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5111.320927] LustreError: 186230:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5111.324788] LustreError: 186230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5111.391516] Lustre: server umount lustre-MDT0000 complete [ 5114.390576] LustreError: 183597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346874 with bad export cookie 10391712013006595053 [ 5114.395465] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5114.537652] LustreError: 186433:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5114.540785] LustreError: 186433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5114.798607] Lustre: server umount lustre-MDT0001 complete [ 5123.772615] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5126.476171] Key type lgssc unregistered [ 5126.809326] LNet: 186965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.815085] LNetError: 186965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.826160] LNet: Removed LNI 192.168.204.148@tcp [ 5127.636309] Key type .llcrypt unregistered [ 5127.638211] Key type ._llcrypt unregistered [ 5148.569512] Key type ._llcrypt registered [ 5148.571527] Key type .llcrypt registered [ 5148.696288] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5149.443677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5149.450640] alg: No test for adler32 (adler32-zlib) [ 5150.521830] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5150.770006] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5152.439166] Key type lgssc registered [ 5153.559413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5160.354028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5160.370657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.778232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.274963] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5171.168947] LustreError: 188361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5171.179511] LustreError: 188361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5172.580093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5173.013709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5176.220727] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5180.241464] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5182.856404] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5188.487865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5188.809626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.147759] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5198.890790] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5206.193504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5213.168688] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5223.492969] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5231.371894] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5238.199452] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5247.460891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.468257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5247.479127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5247.493657] Lustre: Skipped 1 previous similar message [ 5250.580058] LustreError: 192021:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5250.705729] Lustre: server umount lustre-OST0000 complete [ 5255.138101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.146060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.160303] Lustre: Skipped 1 previous similar message [ 5255.170634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.697924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.702740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.918601] LustreError: 192222:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5259.925448] LustreError: 192222:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5259.996208] Lustre: server umount lustre-MDT0000 complete [ 5262.819553] LustreError: 188360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.855648] LustreError: 188360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5263.463682] LustreError: 188340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347023 with bad export cookie 14573663618250036820 [ 5263.467707] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.482049] LustreError: 188340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5263.620348] LustreError: 192425:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5263.622960] LustreError: 192425:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5263.823122] Lustre: server umount lustre-MDT0001 complete [ 5271.657688] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5273.659186] Key type lgssc unregistered [ 5273.885086] LNet: 192956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5273.889793] LNetError: 192956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5273.901731] LNet: Removed LNI 192.168.204.148@tcp [ 5274.560728] Key type .llcrypt unregistered [ 5274.563099] Key type ._llcrypt unregistered [ 5292.261666] Key type ._llcrypt registered [ 5292.263452] Key type .llcrypt registered [ 5292.340836] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5292.905286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5292.999198] alg: No test for adler32 (adler32-zlib) [ 5293.989173] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5294.228783] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5295.959721] Key type lgssc registered [ 5296.929850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5303.532734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5303.550433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.142941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5308.805608] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5315.554762] LustreError: 194330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.567871] LustreError: 194330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5315.578917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5315.860454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5319.408200] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5324.115415] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5326.825477] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5331.821815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5332.231975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5337.356877] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5342.408282] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5350.709645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5366.242621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5366.249571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.264645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5366.295311] Lustre: Skipped 1 previous similar message [ 5369.439100] LustreError: 196751:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5369.709274] Lustre: server umount lustre-OST0000 complete [ 5376.486127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5376.493316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5376.494828] Lustre: Skipped 1 previous similar message [ 5376.502737] Lustre: Skipped 2 previous similar messages [ 5379.565662] LustreError: 196952:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5379.591858] LustreError: 196952:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5379.793572] Lustre: server umount lustre-MDT0000 complete [ 5381.609630] LustreError: 196503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.617291] LustreError: 196503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5382.941804] LustreError: 194309:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347143 with bad export cookie 10508731600334025238 [ 5382.952159] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5383.094318] LustreError: 197153:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5383.104700] LustreError: 197153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5383.332872] Lustre: server umount lustre-MDT0001 complete [ 5391.641564] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5393.947598] Key type lgssc unregistered [ 5394.205175] LNet: 197686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5394.209664] LNetError: 197686:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5394.225225] LNet: Removed LNI 192.168.204.148@tcp [ 5394.842548] Key type .llcrypt unregistered [ 5394.845959] Key type ._llcrypt unregistered [ 5408.703786] Key type ._llcrypt registered [ 5408.706034] Key type .llcrypt registered [ 5408.833821] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5409.577381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5409.611052] alg: No test for adler32 (adler32-zlib) [ 5410.586379] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5410.767612] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5412.440360] Key type lgssc registered [ 5413.246254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5418.902555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5418.915205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.228981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.948082] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5429.526889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5429.848308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5433.002927] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5437.726265] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5440.237478] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5445.652725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5445.996840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5447.021480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5450.439375] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5454.386707] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5464.550717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.558580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5464.576188] Lustre: Skipped 1 previous similar message [ 5464.587342] Lustre: Skipped 1 previous similar message [ 5469.668254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.670649] Lustre: Skipped 1 previous similar message [ 5470.265241] LustreError: 201307:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5470.336438] Lustre: server umount lustre-OST0000 complete [ 5474.787366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.797893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5474.808830] Lustre: Skipped 1 previous similar message [ 5479.092174] LustreError: 201508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5479.099089] LustreError: 201508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5479.177071] Lustre: server umount lustre-MDT0000 complete [ 5479.912627] LustreError: 198911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.926531] LustreError: 198911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5481.811582] LustreError: 198891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347242 with bad export cookie 16855131036104060236 [ 5481.816909] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5481.820141] LustreError: 198891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5481.927737] LustreError: 201711:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5481.930380] LustreError: 201711:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5482.130659] Lustre: server umount lustre-MDT0001 complete [ 5489.780292] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5491.599750] Key type lgssc unregistered [ 5491.787687] LNet: 202243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5491.792981] LNetError: 202243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5491.805268] LNet: Removed LNI 192.168.204.148@tcp [ 5492.352905] Key type .llcrypt unregistered [ 5492.354304] Key type ._llcrypt unregistered [ 5504.989468] Key type ._llcrypt registered [ 5504.990875] Key type .llcrypt registered [ 5505.060850] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5505.730785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5505.751753] alg: No test for adler32 (adler32-zlib) [ 5506.676572] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5506.823958] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5508.447120] Key type lgssc registered [ 5509.253729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5515.244123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5515.258346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5516.551667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.891376] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5524.954185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5525.339759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5527.815444] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5531.047294] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5533.280957] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5537.618460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5537.843168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.906912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5541.620488] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5544.839564] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5550.698962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5550.749583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5550.897701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5554.903160] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5558.825455] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5560.154207] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5571.460996] Lustre: Permanently deactivating lustre-OST0001 [ 5571.469826] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5580.353886] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5583.920598] Lustre: Permanently reactivating lustre-OST0001 [ 5583.940718] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5583.946542] Lustre: Skipped 2 previous similar messages [ 5585.723417] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.743457] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5585.753946] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5585.760801] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5585.786102] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5593.572807] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.587217] Lustre: Skipped 1 previous similar message [ 5593.600265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5593.606685] Lustre: Skipped 1 previous similar message [ 5597.897261] LustreError: 206917:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5597.969426] Lustre: server umount lustre-OST0001 complete [ 5598.695763] LustreError: 205069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.704892] LustreError: 205069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5601.252767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.255588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.263827] Lustre: Skipped 1 previous similar message [ 5601.267776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5602.274769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5602.286467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.818146] LustreError: 205066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.824359] LustreError: 205066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5607.042108] LustreError: 207118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5607.056741] LustreError: 207118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5607.184429] Lustre: server umount lustre-OST0000 complete [ 5612.513177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5612.516851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.522405] Lustre: Skipped 1 previous similar message [ 5612.526346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.529427] Lustre: Skipped 2 previous similar messages [ 5615.716426] LustreError: 207320:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5615.718523] LustreError: 207320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5615.793193] Lustre: server umount lustre-MDT0000 complete [ 5618.381726] LustreError: 203450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347378 with bad export cookie 16111907654550410779 [ 5618.390671] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.515300] LustreError: 207521:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5618.518605] LustreError: 207521:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5618.716592] Lustre: server umount lustre-MDT0001 complete [ 5625.283178] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5627.395754] Key type lgssc unregistered [ 5627.642082] LNet: 208053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5627.647583] LNetError: 208053:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5627.663905] LNet: Removed LNI 192.168.204.148@tcp [ 5628.337123] Key type .llcrypt unregistered [ 5628.338981] Key type ._llcrypt unregistered [ 5649.995510] Key type ._llcrypt registered [ 5649.996708] Key type .llcrypt registered [ 5650.061446] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 5659.759726] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5660.455523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5660.470898] alg: No test for adler32 (adler32-zlib) [ 5661.420975] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 5661.579558] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5663.215765] Key type lgssc registered [ 5663.820593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5668.936615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5672.740239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.359852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5679.857725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.090297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.121470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5684.135524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.273222] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5685.294557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5685.340936] Lustre: lustre-MDT0000: new disk, initializing [ 5685.390276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.404787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5687.747884] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5695.243358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5695.290723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5695.345988] Lustre: 211977:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5695.397943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5695.401582] Lustre: Skipped 1 previous similar message [ 5695.476522] Lustre: lustre-MDT0001: new disk, initializing [ 5695.559188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.593543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5695.604799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5697.940733] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5702.565363] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5704.605813] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5708.821753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5708.888582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5709.153574] Lustre: lustre-OST0000: new disk, initializing [ 5709.159970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5709.230773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.010666] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5715.572639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5715.596449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5715.682394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5718.888806] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5722.373659] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5722.554417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5726.361729] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5726.572645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5731.300098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5731.300293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.302482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5731.303800] LustreError: Skipped 1 previous similar message [ 5731.313508] Lustre: Skipped 1 previous similar message [ 5734.325201] LustreError: 214917:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5734.400359] Lustre: server umount lustre-OST0000 complete [ 5741.537778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5741.538192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5741.546743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.553084] Lustre: Skipped 1 previous similar message [ 5742.679506] LustreError: 215118:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5742.683167] LustreError: 215118:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5742.739086] Lustre: server umount lustre-MDT0000 complete [ 5745.072050] LustreError: 211967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347505 with bad export cookie 15440501984153119884 [ 5745.075316] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5745.080514] LustreError: 211967:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5745.175513] LustreError: 215319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5745.180693] LustreError: 215319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5745.269442] Lustre: server umount lustre-MDT0001 complete [ 5750.480170] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 21:45:10 (1763347510) [ 5755.469861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.890110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.522703] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5760.994152] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.009669] LustreError: 215989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5765.042385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5765.463543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5768.627165] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5773.187662] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5776.208980] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5782.580453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5782.938400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.704799] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5792.310218] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5804.527115] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5809.806839] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5818.036090] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5825.113252] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5837.056241] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5837.059725] Lustre: Skipped 1 previous similar message [ 5854.266241] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5854.274451] Lustre: Skipped 1 previous similar message [ 5887.254562] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5887.257949] Lustre: Skipped 3 previous similar messages [ 5959.087910] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5959.092325] Lustre: Skipped 8 previous similar messages [ 5973.114731] LustreError: 219484:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5973.124908] LustreError: 219484:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5973.200416] Lustre: server umount lustre-OST0000 complete [ 5980.639994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5980.643838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5980.649514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.667036] Lustre: Skipped 2 previous similar messages [ 5985.763832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.768197] Lustre: Skipped 2 previous similar messages [ 5989.855276] Lustre: 210191:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347734/real 1763347734] req@000000006bde1099 x1849003785537024/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763347750 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5989.877132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.888397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.891762] Lustre: Skipped 1 previous similar message [ 5990.367128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5990.451395] LustreError: 219685:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5990.459265] LustreError: 219685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5990.515667] Lustre: server umount lustre-MDT0000 complete [ 5990.879178] Lustre: 210190:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347735/real 1763347735] req@00000000649dba68 x1849003785537792/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763347751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5993.177529] LustreError: 218937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347753 with bad export cookie 15440501984153120766 [ 5993.179734] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.184138] LustreError: 218937:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5993.433832] Lustre: server umount lustre-MDT0001 complete [ 5999.497677] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6000.970533] Key type lgssc unregistered [ 6001.127812] LNet: 220420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6001.131603] LNetError: 220420:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6001.140042] LNet: Removed LNI 192.168.204.148@tcp [ 6001.596967] Key type .llcrypt unregistered [ 6001.598709] Key type ._llcrypt unregistered [ 6016.368476] Key type ._llcrypt registered [ 6016.372458] Key type .llcrypt registered [ 6016.449453] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6016.893360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6016.913271] alg: No test for adler32 (adler32-zlib) [ 6017.844069] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 6017.975960] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6019.592332] Key type lgssc registered [ 6020.305282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6025.536091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6025.550417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.917814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.924072] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6035.784764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6036.041653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.189279] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6042.096966] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6044.671583] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6049.994251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6050.376824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.601930] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6058.395799] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6072.296289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6072.303986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6072.329706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6072.351636] Lustre: Skipped 1 previous similar message [ 6074.704457] LustreError: 224187:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6074.840268] Lustre: server umount lustre-OST0000 complete [ 6082.017179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6082.025991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.047913] Lustre: Skipped 1 previous similar message [ 6082.051171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.649213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.657262] Lustre: Skipped 2 previous similar messages [ 6092.255121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.320076] LustreError: 224392:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6092.326902] LustreError: 224392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6092.380895] Lustre: server umount lustre-MDT0000 complete [ 6092.768773] LustreError: 222594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.777710] LustreError: 222594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6094.668351] LustreError: 221772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347854 with bad export cookie 12281862781939638974 [ 6094.671839] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6094.681064] LustreError: 221772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6094.782708] LustreError: 224595:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6094.789229] LustreError: 224595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6094.936888] Lustre: server umount lustre-MDT0001 complete [ 6102.616836] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6104.771747] Key type lgssc unregistered [ 6105.026786] LNet: 225135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6105.036977] LNetError: 225135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6105.049853] LNet: Removed LNI 192.168.204.148@tcp [ 6105.723417] Key type .llcrypt unregistered [ 6105.725271] Key type ._llcrypt unregistered [ 6121.651717] Key type ._llcrypt registered [ 6121.652821] Key type .llcrypt registered [ 6121.750713] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6121.948561] Key type .llcrypt unregistered [ 6121.950078] Key type ._llcrypt unregistered [ 6140.501150] Key type ._llcrypt registered [ 6140.502561] Key type .llcrypt registered [ 6140.560210] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6151.431875] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6152.286856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6152.479076] alg: No test for adler32 (adler32-zlib) [ 6153.535781] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 6153.739651] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6155.463357] Key type lgssc registered [ 6156.362866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6163.674347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6169.727868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6175.850562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6182.185050] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6183.847277] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5932 sec ========== 21:52:23 (1763347943) [ 6185.303466] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:52:24 (1763347944) === [ 6188.231251] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:52:27 (1763347947) === [ 6218.704471] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6220.836610] Key type lgssc unregistered [ 6221.061388] LNet: 231226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6221.067372] LNetError: 231226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6221.080520] LNet: Removed LNI 192.168.204.148@tcp [ 6221.537313] Key type .llcrypt unregistered [ 6221.539071] Key type ._llcrypt unregistered