[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490550712 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002385] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004030] kvm-guest: setup PV IPIs [ 0.006890] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011061] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.020783] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024787] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028138] *** VALIDATE tmpfs *** [ 0.029482] *** VALIDATE proc *** [ 0.030000] *** VALIDATE cgroup *** [ 0.030019] *** VALIDATE cgroup2 *** [ 0.031248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.038000] debug: unmapping init [mem 0xffffffffb8a59000-0xffffffffb8a60fff] [ 0.039000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040026] ... version: 2 [ 0.041012] ... bit width: 48 [ 0.042012] ... generic registers: 4 [ 0.043013] ... value mask: 0000ffffffffffff [ 0.044020] ... max period: 00007fffffffffff [ 0.045019] ... fixed-purpose events: 3 [ 0.046015] ... event mask: 000000070000000f [ 0.047303] rcu: Hierarchical SRCU implementation. [ 0.049521] smp: Bringing up secondary CPUs ... [ 0.050636] x86: Booting SMP configuration: [ 0.051030] .... node #0, CPUs: #1 #2 #3 [ 0.062747] smp: Brought up 1 node, 4 CPUs [ 0.064029] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.259033] node 0 deferred pages initialised in 189ms [ 0.263011] devtmpfs: initialized [ 0.264337] x86/mm: Memory block size: 128MB [ 0.267956] gcov: version magic: 0x41383552 [ 0.271349] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.274097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.276403] pinctrl core: initialized pinctrl subsystem [ 0.278233] [ 0.278824] ************************************************************* [ 0.281021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284019] ** ** [ 0.286017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.288019] ** ** [ 0.291016] ** This means that this kernel is built to expose internal ** [ 0.293027] ** IOMMU data structures, which may compromise security on ** [ 0.295019] ** your system. ** [ 0.298020] ** ** [ 0.300017] ** If you see this message and you are not debugging the ** [ 0.303017] ** kernel, report this immediately to your vendor! ** [ 0.305016] ** ** [ 0.308015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.310015] ************************************************************* [ 0.313104] NET: Registered protocol family 16 [ 0.314576] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.316069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.319085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.323393] cpuidle: using governor menu [ 0.324698] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.327716] PCI: Using configuration type 1 for base access [ 0.329148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.339139] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.341025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.344091] cryptd: max_cpu_qlen set to 1000 [ 0.347301] ACPI: Added _OSI(Module Device) [ 0.349021] ACPI: Added _OSI(Processor Device) [ 0.351023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.352016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.356554] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.363628] ACPI: Interpreter enabled [ 0.365073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.366016] ACPI: Using IOAPIC for interrupt routing [ 0.368473] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372502] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.383619] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.385063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.388044] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.391119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.396505] acpiphp: Slot [2] registered [ 0.399217] acpiphp: Slot [5] registered [ 0.400243] acpiphp: Slot [6] registered [ 0.402204] acpiphp: Slot [7] registered [ 0.403186] acpiphp: Slot [8] registered [ 0.404170] acpiphp: Slot [9] registered [ 0.405000] acpiphp: Slot [10] registered [ 0.407181] acpiphp: Slot [3] registered [ 0.408600] acpiphp: Slot [4] registered [ 0.410131] acpiphp: Slot [11] registered [ 0.411126] acpiphp: Slot [12] registered [ 0.413097] acpiphp: Slot [13] registered [ 0.414111] acpiphp: Slot [14] registered [ 0.415137] acpiphp: Slot [15] registered [ 0.417144] acpiphp: Slot [16] registered [ 0.418111] acpiphp: Slot [17] registered [ 0.419104] acpiphp: Slot [18] registered [ 0.421143] acpiphp: Slot [19] registered [ 0.422101] acpiphp: Slot [20] registered [ 0.424158] acpiphp: Slot [21] registered [ 0.425149] acpiphp: Slot [22] registered [ 0.427146] acpiphp: Slot [23] registered [ 0.428168] acpiphp: Slot [24] registered [ 0.429082] acpiphp: Slot [25] registered [ 0.430100] acpiphp: Slot [26] registered [ 0.431144] acpiphp: Slot [27] registered [ 0.433158] acpiphp: Slot [28] registered [ 0.434153] acpiphp: Slot [29] registered [ 0.436178] acpiphp: Slot [30] registered [ 0.437167] acpiphp: Slot [31] registered [ 0.439097] PCI host bridge to bus 0000:00 [ 0.441026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.443049] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.445037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.450036] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.454054] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.456250] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.459123] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.461320] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.470024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.474914] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.478073] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.481025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.483027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485702] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.488107] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.491060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.492953] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.498023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.510022] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.515103] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.520199] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.527021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.534023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.546024] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.555694] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.569023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584019] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.593181] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.603021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.615021] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.624869] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.630015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.635018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.647021] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.657184] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.664043] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.681023] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.691254] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.698019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.704022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.715018] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.726943] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.729342] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.731311] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740116] iommu: Default domain type: Passthrough [ 0.741000] SCSI subsystem initialized [ 0.741193] ACPI: bus type USB registered [ 0.742231] usbcore: registered new interface driver usbfs [ 0.744463] usbcore: registered new interface driver hub [ 0.747119] usbcore: registered new device driver usb [ 0.749183] pps_core: LinuxPPS API ver. 1 registered [ 0.750065] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754105] PTP clock support registered [ 0.759113] EDAC MC: Ver: 3.0.0 [ 0.760182] PCI: Using ACPI for IRQ routing [ 0.761000] NetLabel: Initializing [ 0.761009] NetLabel: domain hash size = 128 [ 0.762000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.764116] NetLabel: unlabeled traffic allowed by default [ 0.766196] vgaarb: loaded [ 0.769206] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775000] clocksource: Switched to clocksource kvm-clock [ 0.879765] VFS: Disk quotas dquot_6.6.0 [ 0.881370] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.883891] *** VALIDATE ramfs *** [ 0.885204] *** VALIDATE hugetlbfs *** [ 0.886616] pnp: PnP ACPI init [ 0.888756] pnp: PnP ACPI: found 6 devices [ 0.903238] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.907254] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.909921] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912634] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915415] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.917731] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.921132] NET: Registered protocol family 2 [ 0.923847] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.929099] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932293] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937191] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.940523] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942706] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944918] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.947403] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949829] NET: Registered protocol family 1 [ 0.952175] RPC: Registered named UNIX socket transport module. [ 0.954013] RPC: Registered udp transport module. [ 0.955412] RPC: Registered tcp transport module. [ 0.956701] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.958452] NET: Registered protocol family 44 [ 0.959755] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.962491] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.965104] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.967744] PCI: CLS 0 bytes, default 64 [ 0.970483] Unpacking initramfs... [ 2.404791] debug: unmapping init [mem 0xffff8d5ffcc54000-0xffff8d5ffffbffff] [ 2.408886] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411169] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414880] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.908997] Initialise system trusted keyrings [ 2.910726] Key type blacklist registered [ 2.912696] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.921550] zbud: loaded [ 2.924686] *** VALIDATE nfs *** [ 2.925994] *** VALIDATE nfs4 *** [ 2.927718] pstore: using deflate compression [ 2.930780] Platform Keyring initialized [ 3.032039] NET: Registered protocol family 38 [ 3.033449] Key type asymmetric registered [ 3.035796] Asymmetric key parser 'x509' registered [ 3.038085] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.040907] io scheduler mq-deadline registered [ 3.042641] io scheduler kyber registered [ 3.043977] io scheduler bfq registered [ 3.046521] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.050137] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.053415] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.056989] ACPI: Power Button [PWRF] [ 3.145877] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.236073] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.416347] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.505687] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.696540] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.724434] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.752189] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.757275] Non-volatile memory driver v1.3 [ 3.758780] Linux agpgart interface v0.103 [ 3.795096] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 3.798884] vda: detected capacity change from 0 to 68423680 [ 3.820199] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.822820] vdb: detected capacity change from 0 to 1073741824 [ 3.844517] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.847133] vdc: detected capacity change from 0 to 2621440000 [ 3.862377] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.864983] vdd: detected capacity change from 0 to 2621440000 [ 3.879134] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.882232] vde: detected capacity change from 0 to 4294967296 [ 3.898460] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.900681] vdf: detected capacity change from 0 to 4294967296 [ 3.907451] libphy: Fixed MDIO Bus: probed [ 3.915421] usbcore: registered new interface driver usbserial_generic [ 3.917887] usbserial: USB Serial support registered for generic [ 3.921282] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.925620] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.927707] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.931062] mousedev: PS/2 mouse device common for all mice [ 3.933832] rtc_cmos 00:05: RTC can wake from S4 [ 3.936691] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.938259] rtc_cmos 00:05: registered as rtc0 [ 3.943854] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.944163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.946399] intel_pstate: CPU model not supported [ 3.952360] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.956653] hid: raw HID events driver (C) Jiri Kosina [ 3.959321] usbcore: registered new interface driver usbhid [ 3.961814] usbhid: USB HID core driver [ 3.963873] drop_monitor: Initializing network drop monitor service [ 3.966516] Initializing XFRM netlink socket [ 3.968830] NET: Registered protocol family 10 [ 3.973272] Segment Routing with IPv6 [ 3.974607] NET: Registered protocol family 17 [ 3.977502] mpls_gso: MPLS GSO support [ 3.983703] RAS: Correctable Errors collector initialized. [ 3.986401] AVX version of gcm_enc/dec engaged. [ 3.988198] AES CTR mode by8 optimization enabled [ 4.064143] sched_clock: Marking stable (4064119168, 0)->(5073223771, -1009104603) [ 4.067885] registered taskstats version 1 [ 4.073115] Loading compiled-in X.509 certificates [ 4.075490] zswap: loaded using pool lzo/zbud [ 4.104819] Key type big_key registered [ 4.119488] Key type encrypted registered [ 4.121239] ima: No TPM chip found, activating TPM-bypass! [ 4.122986] ima: Allocated hash algorithm: sha1 [ 4.124450] ima: No architecture policies found [ 4.126334] evm: Initialising EVM extended attributes: [ 4.128332] evm: security.selinux [ 4.129595] evm: security.ima [ 4.130765] evm: security.capability [ 4.132217] evm: HMAC attrs: 0x1 [ 4.134814] rtc_cmos 00:05: setting system clock to 2025-11-14 20:47:27 UTC (1763153247) [ 4.141044] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 4.144106] debug: unmapping init [mem 0xffffffffb8782000-0xffffffffb8a58fff] [ 4.153104] Write protecting the kernel read-only data: 28672k [ 4.156234] debug: unmapping init [mem 0xffffffffb6e03000-0xffffffffb6ffffff] [ 4.159163] debug: unmapping init [mem 0xffffffffb7714000-0xffffffffb77fffff] [ 4.197641] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.207144] systemd[1]: Detected virtualization kvm. [ 4.209273] systemd[1]: Detected architecture x86-64. [ 4.211649] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.242042] systemd[1]: No hostname configured. [ 4.243892] systemd[1]: Set hostname to . [ 4.246041] random: systemd: uninitialized urandom read (16 bytes read) [ 4.248960] systemd[1]: Initializing machine ID from random generator. [ 4.300508] random: ln: uninitialized urandom read (6 bytes read) [ 4.393682] random: systemd: uninitialized urandom read (16 bytes read) [ 4.397058] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.405091] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.410555] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.066760] device-mapper: uevent: version 1.0.3 [ 5.069173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK [ 5.695119] random: fast init done ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.721284] virtio_net virtio0 ens2: renamed from eth0 [ 5.737021] scsi host0: ata_piix [ 5.773625] scsi host1: ata_piix [ 5.789312] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.792513] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.036363] dracut-initqueue[595]: RTNETLINK answers: File exists [ 10.637232] random: crng init done [ 10.638504] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.956131] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.063083] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.305179] SELinux: Disabled at runtime. [ 12.366116] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.375548] systemd[1]: Detected virtualization kvm. [ 12.377644] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.819478] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.823114] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.827568] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.830890] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.834234] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.841702] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.846082] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.900362] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 13.228817] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.518378] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.545345] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.657113] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.669063] EDAC sbridge: Ver: 1.1.2 [ 15.269205] Key type dns_resolver registered [ 15.563158] NFS: Registering the id_resolver key type [ 15.565082] Key type id_resolver registered [ 15.566744] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg101-server login: [ 40.062435] libcfs: loading out-of-tree module taints kernel. [ 40.075147] Key type ._llcrypt registered [ 40.076687] Key type .llcrypt registered [ 40.119849] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 49.821670] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 50.796440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.808486] alg: No test for adler32 (adler32-zlib) [ 51.968490] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 52.680677] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 54.367196] Key type lgssc registered [ 55.690233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.229341] hrtimer: interrupt took 6329405 ns [ 71.523221] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 73.655387] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.138974] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.772344] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.257523] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 100.941667] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 109.170190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.205661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 109.215465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 110.354043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 110.374577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 110.427905] Lustre: lustre-MDT0000: new disk, initializing [ 110.495396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 110.512166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 112.914792] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.551584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 122.614654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 122.668314] Lustre: 6502:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 122.707214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 122.709942] Lustre: Skipped 1 previous similar message [ 122.828284] Lustre: lustre-MDT0001: new disk, initializing [ 122.911710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 122.944073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 122.957438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 125.694616] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.944827] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 134.967802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.023782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 135.221383] Lustre: lustre-OST0000: new disk, initializing [ 135.227976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 135.288539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.817476] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.848210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 140.873427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 140.990906] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 153.622550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.689634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 153.793257] Lustre: lustre-OST0001: new disk, initializing [ 153.797306] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 153.840595] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.390828] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.795789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 160.800996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 160.886556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 168.135747] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 172.857720] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 181.721422] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing check_logdir /tmp/testlogs/ [ 184.513183] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing yml_node [ 187.638511] Lustre: DEBUG MARKER: Client: 2.16.61.2 [ 189.217623] Lustre: DEBUG MARKER: MDS: 2.16.61.2 [ 190.824397] Lustre: DEBUG MARKER: OSS: 2.16.61.2 [ 191.957537] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Nov 14 15:50:34 EST 2025 [ 205.050663] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 206.204736] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 220.131341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 220.135259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 220.142267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.176757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 222.178831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 222.186588] Lustre: Skipped 1 previous similar message [ 222.192071] Lustre: Skipped 1 previous similar message [ 225.958767] LustreError: 12207:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 226.029958] Lustre: server umount lustre-MDT0000 complete [ 228.531080] LustreError: 6493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153471 with bad export cookie 6139602966486514358 [ 228.533257] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 228.537688] LustreError: 6493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 232.415909] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.419099] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.438222] Lustre: Skipped 1 previous similar message [ 232.439686] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 232.451693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 232.455153] Lustre: Skipped 1 previous similar message [ 233.439636] LustreError: 10027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 233.455221] LustreError: 10027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 234.702677] LustreError: 12410:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 234.706495] LustreError: 12410:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 234.792498] Lustre: server umount lustre-MDT0001 complete [ 237.389168] LustreError: 12611:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 237.400077] LustreError: 12611:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 237.439996] Lustre: server umount lustre-OST0000 complete [ 240.110637] LustreError: 12811:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 240.121804] LustreError: 12811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 240.197636] Lustre: server umount lustre-OST0001 complete [ 244.407796] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 249.935245] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 256.410177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 260.674195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 264.722395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 268.950468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 273.293477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 273.343405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 273.479318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 273.501852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 273.554746] Lustre: lustre-MDT0000: new disk, initializing [ 273.618890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.632175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 275.946590] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 282.524377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 282.564701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 282.618572] Lustre: 14992:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 282.656568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 282.659754] Lustre: Skipped 1 previous similar message [ 282.708311] Lustre: lustre-MDT0001: new disk, initializing [ 282.764210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 282.792282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 282.799658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 284.956489] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 290.456671] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.276939] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 298.364940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 298.424841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 300.581698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 300.594880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 300.661518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 302.749446] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 308.509980] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 311.291939] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 311.453591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 314.084674] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 314.248789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 315.875034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.876183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 315.883201] Lustre: Skipped 1 previous similar message [ 315.887372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 315.893015] Lustre: Skipped 3 previous similar messages [ 320.994117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 321.601778] LustreError: 17930:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 321.607254] LustreError: 17930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 321.712404] Lustre: server umount lustre-OST0000 complete [ 326.117770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.779201] LustreError: 18131:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 330.781314] LustreError: 18131:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 330.842752] Lustre: server umount lustre-MDT0000 complete [ 331.235116] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.242087] LustreError: 15004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 333.154601] LustreError: 14985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153576 with bad export cookie 6139602966486515891 [ 333.157471] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.160852] LustreError: 14985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.338716] Lustre: server umount lustre-MDT0001 complete [ 335.279521] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:52:58 (1763153578) [ 339.298938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.602630] LustreError: 18861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.645525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.649228] Lustre: Skipped 1 previous similar message [ 342.034755] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 345.056629] LustreError: 18861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.192186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 351.007513] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 354.433717] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 356.490221] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 362.249353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 368.243345] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 373.415975] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 380.384194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 380.387191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.420359] Lustre: Skipped 1 previous similar message [ 380.428498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 380.431156] Lustre: Skipped 3 previous similar messages [ 383.461664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 385.787964] LustreError: 21107:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 385.794270] LustreError: 21107:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 385.858804] Lustre: server umount lustre-OST0000 complete [ 389.604739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 395.108410] Lustre: server umount lustre-MDT0000 complete [ 398.060097] LustreError: 18840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153641 with bad export cookie 6139602966486516773 [ 398.062775] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.065134] LustreError: 18840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 398.262960] Lustre: server umount lustre-MDT0001 complete [ 407.626879] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 409.930454] Key type lgssc unregistered [ 410.192766] LNet: 22043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.207448] LNetError: 22043:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 410.230504] LNet: Removed LNI 192.168.201.101@tcp [ 410.911164] Key type .llcrypt unregistered [ 410.913713] Key type ._llcrypt unregistered [ 427.446123] Key type ._llcrypt registered [ 427.451282] Key type .llcrypt registered [ 427.541905] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 428.244046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 428.266910] alg: No test for adler32 (adler32-zlib) [ 429.311424] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 429.476771] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 431.119169] Key type lgssc registered [ 432.084887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 437.881474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 437.912073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.268739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.161221] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 448.029433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.305561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.861405] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 454.890812] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 457.710653] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 463.079239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 463.290639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.325567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 467.866644] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 472.128235] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 483.810839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.811894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.812771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.816730] Lustre: Skipped 1 previous similar message [ 488.929421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.933070] Lustre: Skipped 2 previous similar messages [ 489.012126] LustreError: 26101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 489.088083] Lustre: server umount lustre-OST0000 complete [ 494.052890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.054899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 494.081518] Lustre: Skipped 1 previous similar message [ 497.614299] LustreError: 26303:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 497.617712] LustreError: 26303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 497.684214] Lustre: server umount lustre-MDT0000 complete [ 499.178144] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.187469] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 500.271943] LustreError: 23396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153743 with bad export cookie 11408671826842307784 [ 500.273882] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.289180] LustreError: 23396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 500.399267] LustreError: 26505:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 500.402757] LustreError: 26505:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 500.519701] Lustre: server umount lustre-MDT0001 complete [ 508.000626] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 509.918582] Key type lgssc unregistered [ 510.177193] LNet: 27036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.184087] LNetError: 27036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.197925] LNet: Removed LNI 192.168.201.101@tcp [ 510.779988] Key type .llcrypt unregistered [ 510.781969] Key type ._llcrypt unregistered [ 525.422513] Key type ._llcrypt registered [ 525.425743] Key type .llcrypt registered [ 525.490334] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 526.019642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 526.037183] alg: No test for adler32 (adler32-zlib) [ 526.965222] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 527.095134] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 528.719160] Key type lgssc registered [ 529.348054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.096620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 534.112655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.394869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.425948] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 541.957736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 542.168738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 544.205903] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 547.607837] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.092685] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.125216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 560.492260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.535658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 565.025228] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 568.644830] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.993244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.999269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 577.022302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.020769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.022442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.029594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.722481] LustreError: 30996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 579.787930] Lustre: server umount lustre-OST0000 complete [ 583.138097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.138269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 583.139355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.139360] Lustre: Skipped 1 previous similar message [ 583.146367] Lustre: Skipped 1 previous similar message [ 588.262808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.270762] Lustre: Skipped 2 previous similar messages [ 588.695449] LustreError: 31197:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 588.697702] LustreError: 31197:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 588.751109] Lustre: server umount lustre-MDT0000 complete [ 591.388737] LustreError: 28387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153834 with bad export cookie 14179369441908740644 [ 591.394116] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.397854] LustreError: 28387:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 591.555356] LustreError: 31398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 591.563779] LustreError: 31398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 591.688957] Lustre: server umount lustre-MDT0001 complete [ 598.517891] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 600.585655] Key type lgssc unregistered [ 600.810786] LNet: 31930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.822090] LNetError: 31930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.898090] LNet: Removed LNI 192.168.201.101@tcp [ 602.453780] Key type .llcrypt unregistered [ 602.456152] Key type ._llcrypt unregistered [ 620.440757] Key type ._llcrypt registered [ 620.442554] Key type .llcrypt registered [ 620.624861] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 621.294306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 621.321199] alg: No test for adler32 (adler32-zlib) [ 622.311918] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 622.493844] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 624.175150] Key type lgssc registered [ 625.114209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 632.652479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.687782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.124276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.129783] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 644.580614] LustreError: 33302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.589187] LustreError: 33302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 645.451706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 645.719361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 648.157803] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 651.769335] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 654.634407] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 659.107899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 659.333301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.272281] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 667.400698] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 678.779620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 689.121328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.135331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.156541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 692.195780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 692.196507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 692.201039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 692.227049] Lustre: Skipped 1 previous similar message [ 694.408585] LustreError: 35573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 694.507147] Lustre: server umount lustre-OST0000 complete [ 697.317492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.321224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.325573] Lustre: Skipped 1 previous similar message [ 702.434773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 702.439557] Lustre: Skipped 2 previous similar messages [ 703.333067] LustreError: 35774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 703.338265] LustreError: 35774:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 703.396464] Lustre: server umount lustre-MDT0000 complete [ 706.239761] LustreError: 33284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763153949 with bad export cookie 17508574399635557122 [ 706.246071] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 706.248038] LustreError: 33284:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 706.379471] LustreError: 35976:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 706.382738] LustreError: 35976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 706.500967] Lustre: server umount lustre-MDT0001 complete [ 713.692633] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 715.831555] Key type lgssc unregistered [ 716.151312] LNet: 36507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.157349] LNetError: 36507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.171828] LNet: Removed LNI 192.168.201.101@tcp [ 716.863598] Key type .llcrypt unregistered [ 716.869171] Key type ._llcrypt unregistered [ 737.481973] Key type ._llcrypt registered [ 737.485326] Key type .llcrypt registered [ 737.616376] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 738.747742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.796869] alg: No test for adler32 (adler32-zlib) [ 739.782861] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 739.943345] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 741.575197] Key type lgssc registered [ 742.623064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.944723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.961240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.340451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.191714] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 757.944796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 758.187847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 761.024276] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 765.465915] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 768.410484] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 774.077181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 774.360281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.417220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 778.662250] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 782.803697] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 788.966362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.968458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.976619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 788.989272] Lustre: Skipped 1 previous similar message [ 790.289348] Lustre: lustre-OST0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 790.293332] Lustre: Skipped 1 previous similar message [ 794.092658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 794.104366] Lustre: Skipped 1 previous similar message [ 799.201737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.204590] Lustre: Skipped 2 previous similar messages [ 803.295082] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 803.345284] LustreError: 40140:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 803.466547] Lustre: server umount lustre-OST0000 complete [ 824.800615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.812126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 824.825570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 824.825581] Lustre: Skipped 1 previous similar message [ 826.981438] LustreError: 40533:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 827.005762] LustreError: 40533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 827.140619] Lustre: server umount lustre-MDT0000 complete [ 828.962241] LustreError: 37895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 828.985908] LustreError: 37895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 830.308538] LustreError: 37877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154073 with bad export cookie 7764094354080095760 [ 830.317114] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 830.347886] LustreError: 37877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 836.449981] LustreError: 40735:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 836.453817] LustreError: 40735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 836.557994] Lustre: server umount lustre-MDT0001 complete [ 843.068253] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 844.986809] Key type lgssc unregistered [ 845.258055] LNet: 41267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.264705] LNetError: 41267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.275572] LNet: Removed LNI 192.168.201.101@tcp [ 845.824692] Key type .llcrypt unregistered [ 845.828141] Key type ._llcrypt unregistered [ 862.705922] Key type ._llcrypt registered [ 862.708455] Key type .llcrypt registered [ 862.833770] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 863.695241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 863.774818] alg: No test for adler32 (adler32-zlib) [ 864.829693] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 865.027813] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 866.671214] Key type lgssc registered [ 867.616235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 873.838575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 873.857941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.214413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 878.287212] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 884.725148] LustreError: 42638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.748675] LustreError: 42638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 885.448893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 885.742763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 889.053629] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 893.480147] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 896.799236] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 902.620773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 902.833235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.421257] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 910.378851] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 920.245641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 926.485723] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.1@tcp (stopping) [ 926.688597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 926.716273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.199947] LustreError: 44919:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 928.328859] Lustre: server umount lustre-MDT0000 complete [ 930.787177] LustreError: 43350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 930.788956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.040543] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154174 with bad export cookie 16565155701676930649 [ 931.042351] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 931.047075] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 931.174849] LustreError: 45122:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 931.179066] LustreError: 45122:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 931.261950] Lustre: server umount lustre-MDT0001 complete [ 942.852782] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.1@tcp arrived at 1763154186 with bad export cookie 16565155701676931286 [ 942.873800] LustreError: 42619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 961.152977] LustreError: 45344:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 961.163219] LustreError: 45344:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 961.318192] Lustre: server umount lustre-OST0000 complete [ 974.083793] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 976.368359] Key type lgssc unregistered [ 976.677191] LNet: 46183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.690717] LNetError: 46183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.716688] LNet: Removed LNI 192.168.201.101@tcp [ 977.459781] Key type .llcrypt unregistered [ 977.462641] Key type ._llcrypt unregistered [ 997.883413] Key type ._llcrypt registered [ 997.884837] Key type .llcrypt registered [ 997.982957] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 998.703064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.720253] alg: No test for adler32 (adler32-zlib) [ 999.765227] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 999.949902] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1001.607392] Key type lgssc registered [ 1002.555352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.833451] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.851554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1024.415427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1029.535322] LustreError: 47530:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 1056.159228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.885585] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1065.721554] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1231.940972] LustreError: 48238:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1232.045584] Lustre: server umount lustre-OST0000 complete [ 1244.154445] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1246.337571] Key type lgssc unregistered [ 1246.615560] LNet: 49083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.619118] LNetError: 49083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.636353] LNet: Removed LNI 192.168.201.101@tcp [ 1247.246577] Key type .llcrypt unregistered [ 1247.251273] Key type ._llcrypt unregistered [ 1265.534405] Key type ._llcrypt registered [ 1265.535756] Key type .llcrypt registered [ 1265.610806] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1266.186292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.272028] alg: No test for adler32 (adler32-zlib) [ 1267.265569] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1267.480531] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1269.151169] Key type lgssc registered [ 1270.020680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.196765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.226126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.624882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.065410] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1287.139298] LustreError: 50475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.171672] LustreError: 50475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1288.688799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1289.049797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.509252] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1297.250244] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1300.425624] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1306.870483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1307.293132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1311.807199] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1316.417064] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1319.904166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1319.920252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1319.933456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1320.199057] LustreError: 52699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1320.429967] Lustre: server umount lustre-OST0000 complete [ 1325.030218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1325.035291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.049575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.074751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1327.075675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.566729] LustreError: 52899:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1329.568888] LustreError: 52899:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1329.618837] Lustre: server umount lustre-MDT0000 complete [ 1332.194865] LustreError: 50475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.203189] LustreError: 50475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1332.609548] LustreError: 50436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154575 with bad export cookie 12734200208542264696 [ 1332.620845] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.622546] LustreError: 50436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1332.731840] LustreError: 53101:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1332.736027] LustreError: 53101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1332.865116] Lustre: server umount lustre-MDT0001 complete [ 1342.066401] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1344.958440] Key type lgssc unregistered [ 1345.276859] LNet: 53636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.283223] LNetError: 53636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.304862] LNet: Removed LNI 192.168.201.101@tcp [ 1345.978656] Key type .llcrypt unregistered [ 1345.981860] Key type ._llcrypt unregistered [ 1365.214605] Key type ._llcrypt registered [ 1365.216155] Key type .llcrypt registered [ 1365.315079] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1366.049285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1366.086641] alg: No test for adler32 (adler32-zlib) [ 1367.080097] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1367.337905] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1368.983208] Key type lgssc registered [ 1369.965935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.712993] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1375.729307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1391.263328] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1396.383235] LustreError: 54986:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 1423.135163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.033669] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1431.294453] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1435.855461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.211343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.259541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1439.388265] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1443.308162] LustreError: 55908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.318299] LustreError: 55908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1446.003345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1446.276378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.209682] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1453.858285] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.008958] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1461.732326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.736705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.738245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.741024] Lustre: Skipped 1 previous similar message [ 1464.599198] LustreError: 57259:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1464.721315] Lustre: server umount lustre-OST0000 complete [ 1487.328760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1487.330193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.341384] LustreError: Skipped 1 previous similar message [ 1487.343121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.343128] Lustre: Skipped 1 previous similar message [ 1487.367852] Lustre: Skipped 1 previous similar message [ 1489.561089] LustreError: 57639:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1489.570502] LustreError: 57639:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1489.661263] Lustre: server umount lustre-MDT0000 complete [ 1492.456180] LustreError: 56429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.475122] LustreError: 56429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1492.542204] LustreError: 54995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154735 with bad export cookie 9482937063049788061 [ 1492.544715] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.555752] LustreError: 54995:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1498.716541] LustreError: 57841:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1498.719129] LustreError: 57841:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1498.890336] Lustre: server umount lustre-MDT0001 complete [ 1507.933702] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1509.682162] Key type lgssc unregistered [ 1509.881252] LNet: 58373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.886578] LNetError: 58373:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.906746] LNet: Removed LNI 192.168.201.101@tcp [ 1510.472212] Key type .llcrypt unregistered [ 1510.474555] Key type ._llcrypt unregistered [ 1527.710226] Key type ._llcrypt registered [ 1527.711985] Key type .llcrypt registered [ 1527.773172] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1528.413035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1528.435727] alg: No test for adler32 (adler32-zlib) [ 1529.427119] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1529.585647] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1531.231288] Key type lgssc registered [ 1532.207728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1538.300166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1538.317414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.758382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.908620] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1549.701408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1549.957821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1553.146468] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1558.058676] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1561.074481] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1566.402076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.656361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.073139] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1575.729768] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1586.192591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1595.361871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1595.365166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.406111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.389343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.396613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.399554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.761093] LustreError: 62012:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1597.885560] Lustre: server umount lustre-OST0000 complete [ 1601.505141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1601.513317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.520064] Lustre: Skipped 1 previous similar message [ 1601.527717] Lustre: Skipped 1 previous similar message [ 1606.628911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.632294] Lustre: Skipped 1 previous similar message [ 1607.383195] LustreError: 62215:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1607.388720] LustreError: 62215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1607.493651] Lustre: server umount lustre-MDT0000 complete [ 1610.889345] LustreError: 62013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763154854 with bad export cookie 5408530366328858887 [ 1610.896734] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1610.899117] LustreError: 62013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1611.065360] LustreError: 62416:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1611.068312] LustreError: 62416:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1611.230994] Lustre: server umount lustre-MDT0001 complete [ 1620.492075] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1622.703803] Key type lgssc unregistered [ 1622.946888] LNet: 62948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.952493] LNetError: 62948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.971271] LNet: Removed LNI 192.168.201.101@tcp [ 1623.619939] Key type .llcrypt unregistered [ 1623.623556] Key type ._llcrypt unregistered [ 1653.992387] Key type ._llcrypt registered [ 1653.994148] Key type .llcrypt registered [ 1654.110190] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1654.985257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.002104] alg: No test for adler32 (adler32-zlib) [ 1656.048478] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1656.291839] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1657.982081] Key type lgssc registered [ 1659.204534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1666.695381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1666.730564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.138104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.548602] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1678.313155] LustreError: 64633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.335801] LustreError: 64633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1679.519309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1680.012834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1683.108625] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1687.493272] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1690.216807] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1695.536924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.777190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.908267] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1703.721179] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1713.215019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1720.090342] Lustre: Failing over lustre-MDT0000 [ 1720.288239] LustreError: 66902:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1720.373504] Lustre: server umount lustre-MDT0000 complete [ 1720.601128] LustreError: 64629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.802964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1720.813857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.360072] LustreError: 64634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.362872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.393428] LustreError: 64634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1723.424576] Lustre: Skipped 1 previous similar message [ 1725.688173] LustreError: 64630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.721382] LustreError: 64630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1730.165163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.298602] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1730.393575] Lustre: *** cfs_fail_loc=135, val=0*** [ 1730.446243] LustreError: 67410:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1730.449702] LustreError: 67410:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -2 [ 1730.459568] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1730.459568] [ 1730.464689] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1730.469795] LustreError: 67398:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1730.473483] LustreError: 67398:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1730.477540] LustreError: 67398:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1730.490623] Lustre: server umount lustre-MDT0000 complete [ 1730.492146] LustreError: 67398:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1730.808965] LustreError: 64630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.829134] LustreError: 64630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1736.649877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.771904] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1736.777279] LustreError: 64626:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1736.781823] LustreError: 63942:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b143deef x1848801911402880/t0(0) o250->MGC192.168.201.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1736.806527] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3fc01b1b886621a2 [ 1736.812480] Lustre: MGC192.168.201.101@tcp: Connection restored to 0@lo (at 0@lo) [ 1737.097675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.160497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1740.713834] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1742.068401] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1742.313530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1742.336540] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1742.373162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1747.433081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.439425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.479295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.482139] Lustre: Skipped 1 previous similar message [ 1749.082657] LustreError: 68222:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1749.105223] LustreError: 68222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1749.225568] Lustre: server umount lustre-OST0000 complete [ 1757.664438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1757.665396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.687680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1757.687687] Lustre: Skipped 1 previous similar message [ 1757.718185] Lustre: Skipped 1 previous similar message [ 1762.788356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.798610] Lustre: Skipped 2 previous similar messages [ 1766.883534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.391181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.554302] LustreError: 68423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1767.586266] LustreError: 68423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1767.745696] Lustre: server umount lustre-MDT0000 complete [ 1771.991455] LustreError: 64614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155015 with bad export cookie 4593701424984367522 [ 1771.993443] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.010739] LustreError: 64614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1772.147135] LustreError: 68626:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1772.150853] LustreError: 68626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1772.435741] Lustre: server umount lustre-MDT0001 complete [ 1782.688399] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1785.406153] Key type lgssc unregistered [ 1785.735960] LNet: 69158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1785.754844] LNetError: 69158:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.791379] LNet: Removed LNI 192.168.201.101@tcp [ 1787.573866] Key type .llcrypt unregistered [ 1787.577129] Key type ._llcrypt unregistered [ 1806.929198] Key type ._llcrypt registered [ 1806.930848] Key type .llcrypt registered [ 1807.077684] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1808.224703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.263291] alg: No test for adler32 (adler32-zlib) [ 1809.332958] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1809.636205] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1811.319178] Key type lgssc registered [ 1812.405093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1818.613563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1818.628568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.040532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.623937] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1830.369133] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.416782] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1832.868413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1833.457571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1837.523391] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1843.622954] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1847.135759] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1853.081281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1853.487070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.015433] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1863.096643] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1864.780584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1873.426752] Lustre: Failing over lustre-MDT0000 [ 1873.558943] LustreError: 72781:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1873.685260] Lustre: server umount lustre-MDT0000 complete [ 1874.400888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1874.407435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.418402] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.917651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.922770] Lustre: Skipped 1 previous similar message [ 1880.034926] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.049104] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1881.576056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1881.725486] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.897341] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1881.995300] LustreError: 73290:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1882.000545] LustreError: 73290:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -9 [ 1882.016908] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1882.016908] [ 1882.029298] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1882.046945] LustreError: 73278:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1882.056132] LustreError: 73278:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1882.063473] LustreError: 73278:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1882.087477] Lustre: server umount lustre-MDT0000 complete [ 1882.091333] LustreError: 73278:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1885.155412] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.174815] LustreError: 70531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1887.860555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.969388] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.157210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.201088] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1890.547329] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1891.053537] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1893.354625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1893.377887] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1893.421239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1898.474105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.477240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1898.490688] Lustre: Skipped 1 previous similar message [ 1898.504516] Lustre: Skipped 1 previous similar message [ 1902.708359] LustreError: 74100:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1902.720238] LustreError: 74100:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1902.810591] Lustre: server umount lustre-OST0000 complete [ 1908.707864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1908.709630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.715756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.721153] Lustre: Skipped 1 previous similar message [ 1911.823497] LustreError: 74303:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1911.831353] LustreError: 74303:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1911.915166] Lustre: server umount lustre-MDT0000 complete [ 1913.824722] LustreError: 72692:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.839880] LustreError: 72692:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1914.907051] LustreError: 71252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155158 with bad export cookie 13109994194991630078 [ 1914.913702] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1914.922608] LustreError: 71252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1915.046612] LustreError: 74504:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1915.050145] LustreError: 74504:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1915.184782] Lustre: server umount lustre-MDT0001 complete [ 1924.287749] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1926.633354] Key type lgssc unregistered [ 1926.889513] LNet: 75036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.898496] LNetError: 75036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.914253] LNet: Removed LNI 192.168.201.101@tcp [ 1927.584125] Key type .llcrypt unregistered [ 1927.586224] Key type ._llcrypt unregistered [ 1938.625237] Key type ._llcrypt registered [ 1938.628866] Key type .llcrypt registered [ 1938.728514] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 1939.009336] Key type .llcrypt unregistered [ 1939.011982] Key type ._llcrypt unregistered [ 1958.741640] Key type ._llcrypt registered [ 1958.743784] Key type .llcrypt registered [ 1958.856038] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 1959.555354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.602736] alg: No test for adler32 (adler32-zlib) [ 1960.983587] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1961.253314] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 1962.927178] Key type lgssc registered [ 1964.159313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1971.870509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1971.905636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.495494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1977.430863] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1982.946154] LustreError: 76889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.961262] LustreError: 76889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1985.007617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1985.410964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1989.244753] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 1994.818053] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1998.307970] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.636386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2004.972789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.774551] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2014.322883] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2018.293735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2028.513234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2028.515489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2028.518371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.522173] Lustre: Skipped 1 previous similar message [ 2033.219493] LustreError: 79160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2033.351772] Lustre: server umount lustre-OST0000 complete [ 2036.705934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.716821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.729635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.733827] Lustre: Skipped 1 previous similar message [ 2038.753658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.759713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.876552] LustreError: 79361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2042.893151] LustreError: 79361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2043.021308] Lustre: server umount lustre-MDT0000 complete [ 2043.874125] LustreError: 76889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2043.888023] LustreError: 76889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2045.862104] LustreError: 76869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155289 with bad export cookie 498196741996547525 [ 2045.866647] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.873263] LustreError: 76869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2045.962947] LustreError: 79563:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2045.966241] LustreError: 79563:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2046.092240] Lustre: server umount lustre-MDT0001 complete [ 2054.806949] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2057.152200] Key type lgssc unregistered [ 2057.448883] LNet: 80095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.455309] LNetError: 80095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.466341] LNet: Removed LNI 192.168.201.101@tcp [ 2058.124501] Key type .llcrypt unregistered [ 2058.127324] Key type ._llcrypt unregistered [ 2081.203392] Key type ._llcrypt registered [ 2081.207367] Key type .llcrypt registered [ 2081.270176] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2082.145397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.161637] alg: No test for adler32 (adler32-zlib) [ 2083.184606] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2083.410817] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2085.063173] Key type lgssc registered [ 2086.367891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2094.098692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2094.131187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.475162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.943536] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2104.803880] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.823535] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2106.223345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2106.503880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.724588] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2114.598228] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2118.012462] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2124.834847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2125.218430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.589956] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2134.214202] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.594791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2145.762312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.766113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2145.774725] Lustre: Skipped 1 previous similar message [ 2145.781176] Lustre: Skipped 1 previous similar message [ 2146.691304] LustreError: 83731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2146.777697] Lustre: server umount lustre-OST0000 complete [ 2150.882894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.891525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2150.933241] Lustre: Skipped 1 previous similar message [ 2151.006441] Lustre: Skipped 1 previous similar message [ 2155.823573] LustreError: 83934:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2155.847770] LustreError: 83934:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2155.922532] Lustre: server umount lustre-MDT0000 complete [ 2156.003403] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2156.010303] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2159.013808] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.031584] LustreError: 81487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2159.307878] LustreError: 81467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155402 with bad export cookie 5716002992689166157 [ 2159.313031] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.321969] LustreError: 81467:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2159.439679] LustreError: 84135:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2159.442029] LustreError: 84135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2159.654361] Lustre: server umount lustre-MDT0001 complete [ 2167.864621] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2170.026611] Key type lgssc unregistered [ 2170.281468] LNet: 84667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.293642] LNetError: 84667:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.307721] LNet: Removed LNI 192.168.201.101@tcp [ 2171.979898] Key type .llcrypt unregistered [ 2171.982458] Key type ._llcrypt unregistered [ 2189.799536] Key type ._llcrypt registered [ 2189.801530] Key type .llcrypt registered [ 2189.902539] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2190.448987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.477167] alg: No test for adler32 (adler32-zlib) [ 2191.486871] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2191.635102] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2193.255173] Key type lgssc registered [ 2194.082317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.610761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.647858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.016401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.430764] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2211.299704] LustreError: 86040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.316087] LustreError: 86040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2212.467820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.764780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2216.774896] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2221.803462] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2225.145801] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2230.531398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2230.894787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2237.088331] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2242.496145] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.255959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2258.922588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.938619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.948838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2259.959824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2259.963903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.989994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.001355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.017992] Lustre: Skipped 1 previous similar message [ 2263.606978] LustreError: 88304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2263.713901] Lustre: server umount lustre-OST0000 complete [ 2269.155707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2269.159500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.194865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2273.865845] LustreError: 88508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2273.874571] LustreError: 88508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2274.004274] Lustre: server umount lustre-MDT0000 complete [ 2275.297801] LustreError: 86035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.313042] LustreError: 86035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2277.347516] LustreError: 86020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155520 with bad export cookie 11428993209366846954 [ 2277.368599] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.508169] LustreError: 88709:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2277.533918] LustreError: 88709:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2277.772470] Lustre: server umount lustre-MDT0001 complete [ 2286.491639] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2288.586657] Key type lgssc unregistered [ 2288.800731] LNet: 89241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.814288] LNetError: 89241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.822574] LNet: Removed LNI 192.168.201.101@tcp [ 2289.401653] Key type .llcrypt unregistered [ 2289.404705] Key type ._llcrypt unregistered [ 2307.295226] Key type ._llcrypt registered [ 2307.296939] Key type .llcrypt registered [ 2307.399218] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2307.908933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2307.979379] alg: No test for adler32 (adler32-zlib) [ 2309.034773] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2309.324634] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2311.039163] Key type lgssc registered [ 2311.935686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2318.311099] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2318.330544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2333.919304] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2339.039223] LustreError: 90586:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 2365.792091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.930781] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2376.121371] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2380.404829] LustreError: 91444:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2380.476487] Lustre: server umount lustre-OST0000 complete [ 2387.165368] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:27:09 (1763155629) [ 2392.951294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.393940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.084785] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2398.689910] LustreError: 92133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.705314] LustreError: 92133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2402.796323] LustreError: 92133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.443559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2404.676044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2408.122971] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2413.519937] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2417.037955] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2422.977655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2423.249337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.199886] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2433.704324] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2438.645319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2448.872882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.877440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.891573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.895529] Lustre: Skipped 1 previous similar message [ 2453.441245] LustreError: 94522:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2453.452205] LustreError: 94522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2453.538454] Lustre: server umount lustre-OST0000 complete [ 2459.104590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.109645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.120264] Lustre: Skipped 2 previous similar messages [ 2462.592894] LustreError: 94723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2462.602237] LustreError: 94723:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2462.741220] Lustre: server umount lustre-MDT0000 complete [ 2464.231042] LustreError: 92111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.244156] LustreError: 92111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2465.555259] LustreError: 92095:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155708 with bad export cookie 11772043347489783178 [ 2465.566498] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.655590] LustreError: 94926:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2465.658255] LustreError: 94926:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2465.778551] Lustre: server umount lustre-MDT0001 complete [ 2474.486352] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2476.711665] Key type lgssc unregistered [ 2476.970488] LNet: 95459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.981183] LNetError: 95459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.999848] LNet: Removed LNI 192.168.201.101@tcp [ 2477.804302] Key type .llcrypt unregistered [ 2477.806178] Key type ._llcrypt unregistered [ 2499.576671] Key type ._llcrypt registered [ 2499.578898] Key type .llcrypt registered [ 2499.691741] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2500.718401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.738994] alg: No test for adler32 (adler32-zlib) [ 2501.752222] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2501.967360] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2503.650360] Key type lgssc registered [ 2504.799511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.275804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.293814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.618225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.236594] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2521.684873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2521.978422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2525.518696] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2531.438095] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2534.992078] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2541.522061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2541.948231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.736155] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2554.427255] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2558.334683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2567.648265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.662853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.693043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.674845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.675108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2568.681466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.579214] LustreError: 99118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2573.708577] Lustre: server umount lustre-OST0000 complete [ 2578.401550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2578.412463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.447713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.456775] Lustre: Skipped 1 previous similar message [ 2582.761142] LustreError: 99320:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2582.769130] LustreError: 99320:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2582.940930] Lustre: server umount lustre-MDT0000 complete [ 2584.033787] LustreError: 96852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2584.044580] LustreError: 96852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2586.759554] LustreError: 99523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763155830 with bad export cookie 9882360242611333682 [ 2586.770612] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2586.779274] LustreError: 99523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2586.900514] LustreError: 99522:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2586.917587] LustreError: 99522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2587.123854] Lustre: server umount lustre-MDT0001 complete [ 2597.707157] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 2600.681798] Key type lgssc unregistered [ 2601.050847] LNet: 100055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.061514] LNetError: 100055:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2601.087679] LNet: Removed LNI 192.168.201.101@tcp [ 2602.011948] Key type .llcrypt unregistered [ 2602.013716] Key type ._llcrypt unregistered [ 2620.730912] Key type ._llcrypt registered [ 2620.733255] Key type .llcrypt registered [ 2620.865559] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2622.654463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2622.718345] alg: No test for adler32 (adler32-zlib) [ 2623.821894] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 2624.088568] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 2625.807198] Key type lgssc registered [ 2626.890896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2634.307612] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.328988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2649.887791] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2655.007236] LustreError: 101279:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 2681.695165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.306403] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2692.504147] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2699.418314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2699.709646] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2699.721135] LustreError: 102191:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2699.727652] LustreError: 102191:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2699.732648] LustreError: 102191:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2699.821620] Lustre: server umount lustre-MDT0000 complete [ 2699.828582] LustreError: 102191:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2716.640573] LustreError: 102622:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2716.799688] Lustre: server umount lustre-OST0000 complete [ 2724.853734] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 2734.113389] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2744.548928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2751.209322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2757.566385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2764.239541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.837497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.906250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.164451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2771.196992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2771.252307] Lustre: lustre-MDT0000: new disk, initializing [ 2771.352503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.368917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2775.202426] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2784.689919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2784.741253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2784.833525] Lustre: 104961:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2784.891829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2784.895137] Lustre: Skipped 1 previous similar message [ 2784.941280] Lustre: lustre-MDT0001: new disk, initializing [ 2785.023042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2785.063179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2785.068264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2788.356118] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2795.151285] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.546563] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2804.361993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2804.418935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2804.643731] Lustre: lustre-OST0000: new disk, initializing [ 2804.651217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2804.725952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.683777] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2810.416312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2810.428846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2810.536078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2817.308655] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.946344] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2822.234214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2826.272835] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2826.533324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2831.329533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.331817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.347976] Lustre: Skipped 1 previous similar message [ 2834.564132] LustreError: 107901:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2834.576923] LustreError: 107901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2834.716322] Lustre: server umount lustre-OST0000 complete [ 2841.569196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2841.572236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.593242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2841.603676] Lustre: Skipped 2 previous similar messages [ 2844.795280] LustreError: 108103:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2844.810538] LustreError: 108103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2844.947915] Lustre: server umount lustre-MDT0000 complete [ 2848.425567] LustreError: 104951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156091 with bad export cookie 7368069854397701570 [ 2848.431494] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2848.554269] LustreError: 108304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2848.557300] LustreError: 108304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2848.718600] Lustre: server umount lustre-MDT0001 complete [ 2856.731662] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:34:58 (1763156098) [ 2860.188284] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2875.520049] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 2885.135420] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 2893.996538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2899.620666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2905.433051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2911.073555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.219059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.289718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.521676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2917.557578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2917.653216] Lustre: lustre-MDT0000: new disk, initializing [ 2917.783987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.823293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2921.567449] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2930.391914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2930.457330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.533149] Lustre: 111440:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2930.589922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2930.594095] Lustre: Skipped 1 previous similar message [ 2930.644947] Lustre: lustre-MDT0001: new disk, initializing [ 2930.719152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.750239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2930.762914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2933.709754] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2939.919395] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2942.960242] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2948.554465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2948.614827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2948.795109] Lustre: lustre-OST0000: new disk, initializing [ 2948.798798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2948.877831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.314586] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2956.846617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2956.851952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2956.921427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2959.138923] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2962.343842] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2962.566956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2965.988965] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2966.152761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2972.129304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2972.134398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.149854] Lustre: Skipped 1 previous similar message [ 2972.153808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2972.157060] Lustre: Skipped 1 previous similar message [ 2973.746168] LustreError: 114378:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2973.750099] LustreError: 114378:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2973.859487] Lustre: server umount lustre-OST0000 complete [ 2976.737334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2976.741622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.749394] Lustre: Skipped 1 previous similar message [ 2976.753738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.368477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2982.371111] Lustre: Skipped 2 previous similar messages [ 2982.776251] LustreError: 114580:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2982.780289] LustreError: 114580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2982.852182] Lustre: server umount lustre-MDT0000 complete [ 2985.506636] LustreError: 113185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156228 with bad export cookie 7368069854397702459 [ 2985.509593] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.523104] LustreError: 113185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2985.755463] Lustre: server umount lustre-MDT0001 complete [ 2991.740601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.100170] LustreError: 115257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.110661] LustreError: 115257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2992.173493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.137472] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 2997.218345] LustreError: 115257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3002.095964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3005.407835] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3009.302926] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3012.193394] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3018.162090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.919429] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3027.608659] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3033.487934] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3038.179843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3038.181198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.182787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.194524] Lustre: Skipped 2 previous similar messages [ 3042.562217] LustreError: 117597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3042.565290] LustreError: 117597:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3042.687978] Lustre: server umount lustre-OST0000 complete [ 3048.416606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.421944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3058.658132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3058.661059] Lustre: Skipped 6 previous similar messages [ 3060.191140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3060.360343] Lustre: server umount lustre-MDT0000 complete [ 3063.564439] LustreError: 115239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156306 with bad export cookie 7368069854397703341 [ 3063.575080] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.580411] LustreError: 115239:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3063.850346] Lustre: server umount lustre-MDT0001 complete [ 3071.942688] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3074.593535] Key type lgssc unregistered [ 3074.869970] LNet: 118535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3074.878330] LNetError: 118535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.948538] LNet: Removed LNI 192.168.201.101@tcp [ 3076.676709] Key type .llcrypt unregistered [ 3076.681130] Key type ._llcrypt unregistered [ 3090.344290] Key type ._llcrypt registered [ 3090.345897] Key type .llcrypt registered [ 3090.409521] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 3099.869335] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3100.525183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3100.549302] alg: No test for adler32 (adler32-zlib) [ 3101.536514] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3101.763401] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3103.400927] Key type lgssc registered [ 3104.161094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3109.963991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3114.420538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3118.529604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.155650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.374207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.407268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3127.423594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.558578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3128.583571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3128.639451] Lustre: lustre-MDT0000: new disk, initializing [ 3128.702748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.719459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3131.331876] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3140.062938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3140.142404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3140.220373] Lustre: 121640:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3140.263556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3140.266473] Lustre: Skipped 1 previous similar message [ 3140.330364] Lustre: lustre-MDT0001: new disk, initializing [ 3140.433769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3140.475632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3140.483034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3143.225052] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3148.412480] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3151.368480] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3156.069543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3156.125027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3156.335268] Lustre: lustre-OST0000: new disk, initializing [ 3156.338173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3156.415910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.432642] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3163.682302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3163.687084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3163.791405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3166.470631] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3169.957355] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3170.108433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3173.193880] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3173.357855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3179.491653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3179.493500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3179.497687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3179.502463] Lustre: Skipped 1 previous similar message [ 3184.611946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3184.615229] Lustre: Skipped 1 previous similar message [ 3189.215884] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3189.246133] LustreError: 124578:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3189.353795] Lustre: server umount lustre-OST0000 complete [ 3194.848909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.853170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3194.857897] Lustre: Skipped 1 previous similar message [ 3194.861746] Lustre: Skipped 1 previous similar message [ 3199.970427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.977490] Lustre: Skipped 2 previous similar messages [ 3205.089062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3205.093958] Lustre: Skipped 1 previous similar message [ 3206.111145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3206.174215] LustreError: 124780:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3206.180815] LustreError: 124780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3206.226739] Lustre: server umount lustre-MDT0000 complete [ 3208.263362] LustreError: 121633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156451 with bad export cookie 10710004347862052802 [ 3208.267852] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.272538] LustreError: 121633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3208.358717] LustreError: 124982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3208.361611] LustreError: 124982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3208.440474] Lustre: server umount lustre-MDT0001 complete [ 3214.145065] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 16:40:56 (1763156456) [ 3217.896782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.199422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.428043] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3223.520797] LustreError: 125654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.529890] LustreError: 125654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3225.130306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3225.388963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3227.803194] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3231.335965] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3233.699440] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3235.105392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3235.106321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3235.113960] Lustre: Skipped 1 previous similar message [ 3241.228194] LustreError: 126998:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3241.230409] LustreError: 126998:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3241.315419] Lustre: server umount lustre-MDT0000 complete [ 3241.441860] LustreError: 125651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.466410] LustreError: 125651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3244.270734] LustreError: 125635:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156487 with bad export cookie 10710004347862053684 [ 3244.285853] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3244.545979] Lustre: server umount lustre-MDT0001 complete [ 3251.427882] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 16:41:33 (1763156493) [ 3256.067691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3271.584421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3276.703434] LustreError: 127843:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3302.431437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.516141] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3310.973865] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3312.351425] LustreError: 128508:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3312.354310] LustreError: 128508:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3312.445187] Lustre: server umount lustre-OST0000 complete [ 3318.472443] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 16:42:41 (1763156561) [ 3323.183524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.469731] LustreError: 129178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.528697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3326.165423] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3328.998227] LustreError: 129178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.806929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3334.197111] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3337.654250] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.861360] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3344.471170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3348.603823] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3352.647353] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3362.786011] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3362.791776] LustreError: Skipped 1 previous similar message [ 3362.793863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.802420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.809267] Lustre: Skipped 3 previous similar messages [ 3367.395017] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3367.401112] Lustre: Skipped 1 previous similar message [ 3368.852883] LustreError: 131423:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3368.860236] LustreError: 131423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3368.906440] Lustre: server umount lustre-MDT0000 complete [ 3371.063531] LustreError: 129158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156614 with bad export cookie 10710004347862054349 [ 3371.068727] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3371.072467] LustreError: 129158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3371.267443] Lustre: server umount lustre-MDT0001 complete [ 3382.796726] Lustre: server umount lustre-OST0000 complete [ 3388.002196] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:43:50 (1763156630) [ 3392.143839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.500670] LustreError: 132498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.556067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.559222] Lustre: Skipped 2 previous similar messages [ 3395.240112] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3401.354450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3404.033534] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3407.282312] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.670220] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3414.651799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3417.966758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3418.817028] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3422.188484] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3424.463512] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3424.575558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3426.792084] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3426.923496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3428.320696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.326076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.338217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.340984] Lustre: Skipped 3 previous similar messages [ 3431.395467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3434.234242] LustreError: 135241:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3434.236374] LustreError: 135241:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3434.296867] Lustre: server umount lustre-OST0000 complete [ 3442.481583] Lustre: server umount lustre-MDT0000 complete [ 3444.504335] LustreError: 135444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156687 with bad export cookie 10710004347862055413 [ 3444.505663] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3444.511823] LustreError: 135444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3444.705952] Lustre: server umount lustre-MDT0001 complete [ 3449.783416] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:44:52 (1763156692) [ 3453.920893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3469.407633] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3474.527489] LustreError: 136289:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3500.127367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.135211] Lustre: Skipped 2 previous similar messages [ 3502.985378] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3505.652180] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3509.005212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.246283] LustreError: 136316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.254543] LustreError: 136316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3509.330073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3511.172457] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3515.242622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3515.362487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3515.365281] Lustre: Skipped 7 previous similar messages [ 3517.295970] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3519.843936] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3521.531692] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3523.467469] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3523.591705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3525.719699] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3525.850243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3530.726810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3530.726938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.740745] Lustre: Skipped 4 previous similar messages [ 3533.065196] LustreError: 139064:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3533.067637] LustreError: 139064:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3533.132132] Lustre: server umount lustre-MDT0000 complete [ 3535.344859] LustreError: 136297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156778 with bad export cookie 10710004347862056428 [ 3535.348063] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.350905] LustreError: 136297:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3535.530720] Lustre: server umount lustre-MDT0001 complete [ 3546.858548] Lustre: server umount lustre-OST0000 complete [ 3551.006806] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 16:46:33 (1763156793) [ 3554.570684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3570.079552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3575.199447] LustreError: 140112:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 3603.188559] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3605.634952] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3609.175986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.410951] LustreError: 140139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.422910] LustreError: 140139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3609.479196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3611.241772] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3615.074238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3617.180550] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3620.192193] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3622.088190] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3625.788522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.822966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3625.902733] Lustre: lustre-OST0001: new disk, initializing [ 3625.905079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3628.593477] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3632.847156] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3634.203447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3634.210241] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3634.269340] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3634.956031] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3635.054529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3636.968277] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3637.076452] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3639.267139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.272351] Lustre: Skipped 1 previous similar message [ 3644.086230] LustreError: 143837:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3644.088473] LustreError: 143837:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3644.113744] Lustre: server umount lustre-OST0000 complete [ 3644.389863] LustreError: 140128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.402134] LustreError: 140128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3649.507208] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3649.507800] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3649.512746] Lustre: Skipped 6 previous similar messages [ 3652.195058] Lustre: server umount lustre-OST0001 complete [ 3659.939957] Lustre: server umount lustre-MDT0000 complete [ 3661.528418] LustreError: 142645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156904 with bad export cookie 10710004347862057373 [ 3661.531108] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.532969] LustreError: 142645:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3661.671177] Lustre: server umount lustre-MDT0001 complete [ 3664.048704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3665.731722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3667.315752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3668.905823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3671.540606] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 16:48:34 (1763156914) [ 3672.214567] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3672.885571] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3673.564075] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 16:48:36 (1763156916) [ 3676.238400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.375297] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3676.390984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3676.492407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.495226] Lustre: Skipped 6 previous similar messages [ 3678.204949] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3682.114662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3682.190290] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3682.208615] Lustre: 146078:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3684.330222] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3686.945093] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.648907] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3691.775636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3691.868423] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3694.573306] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3697.021921] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3698.731492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3699.041615] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3699.162290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3700.983345] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3701.079832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3703.778778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.779057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.782366] LustreError: Skipped 2 previous similar messages [ 3703.788492] Lustre: Skipped 6 previous similar messages [ 3708.060365] Lustre: server umount lustre-OST0000 complete [ 3722.634072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3723.813144] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3723.818758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3723.819558] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3723.826577] Lustre: Skipped 1 previous similar message [ 3725.214243] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3727.455781] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3729.279418] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3729.379360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3731.051216] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3731.148902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3732.606293] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff988fc8e6c000.ost_server_uuid 50 [ 3733.205189] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff988fc8e6c000.ost_server_uuid in FULL state after 0 sec [ 3733.270314] LustreError: 146085:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.201.1@tcp) returned error from blocking AST (req@00000000107ae19d x1848803427819008 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000182f89ea/0x94a19435d9350ad4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.1@tcp remote: 0xeb7208b4139875d4 expref: 5 pid: 146083 timeout: 3833 lvb_type: 0 [ 3733.286869] LustreError: lustre-MDT0000: A client on nid 192.168.201.1@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3733.291433] LustreError: 146073:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.1@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000182f89ea/0x94a19435d9350ad4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.1@tcp remote: 0xeb7208b4139875d4 expref: 6 pid: 146083 timeout: 0 lvb_type: 0 [ 3739.109100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3739.113625] LustreError: Skipped 1 previous similar message [ 3740.630813] Lustre: server umount lustre-OST0000 complete [ 3748.261737] Lustre: server umount lustre-MDT0000 complete [ 3749.344298] LustreError: 146089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3749.351124] LustreError: 146089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3749.799444] LustreError: 146070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763156993 with bad export cookie 10710004347862058542 [ 3749.803599] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.805440] LustreError: 146070:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3764.191143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3764.302046] Lustre: server umount lustre-MDT0001 complete [ 3768.483663] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3769.687647] Key type lgssc unregistered [ 3769.824378] LNet: 151564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3769.827336] LNetError: 151564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3769.835305] LNet: Removed LNI 192.168.201.101@tcp [ 3770.177908] Key type .llcrypt unregistered [ 3770.179462] Key type ._llcrypt unregistered [ 3780.134169] Key type ._llcrypt registered [ 3780.135683] Key type .llcrypt registered [ 3780.182104] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3780.543491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3780.573104] alg: No test for adler32 (adler32-zlib) [ 3781.449316] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3781.552564] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3783.143186] Key type lgssc registered [ 3783.594327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3786.799371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3786.807167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.009242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3789.590344] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3794.746785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3794.927648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3796.424376] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3798.621455] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3800.153553] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3802.753099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3802.893092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.163095] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3807.370283] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3813.158582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3820.149881] Lustre: Failing over lustre-MDT0000 [ 3820.330084] LustreError: 155207:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3820.359807] Lustre: server umount lustre-MDT0000 complete [ 3820.512675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3820.516324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.522580] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.529231] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3823.585266] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.586835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.590599] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3823.593716] Lustre: Skipped 1 previous similar message [ 3828.704500] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.709512] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3831.411280] LustreError: 152922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.1@tcp arrived at 1763157074 with bad export cookie 8563081369350136437 [ 3831.419254] LustreError: 152922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3833.825859] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.832568] LustreError: 152941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3834.633260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3834.711915] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.833575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.863037] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3836.577123] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3837.630334] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3837.633634] Lustre: lustre-MDT0000: Denying connection for new client 7a9b7efb-66bf-48b5-9eed-8d08b810e742 (at 192.168.201.1@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3839.976395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3845.091746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.093051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3845.095949] Lustre: Skipped 1 previous similar message [ 3850.210882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.214125] Lustre: Skipped 2 previous similar messages [ 3850.674191] LustreError: 156137:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3850.676975] LustreError: 156137:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3850.737374] Lustre: server umount lustre-OST0000 complete [ 3852.389613] LustreError: 156344:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3852.397072] Lustre: 155709:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3852.400695] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3852.410142] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3852.416859] LustreError: 155709:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009b401e27 x1848804139634688/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3852.429127] LustreError: 155709:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3852.436938] LustreError: 155709:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3852.445468] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 3852.546754] LustreError: 156344:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3852.550184] LustreError: 156344:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3852.592168] Lustre: server umount lustre-MDT0000 complete [ 3854.266762] LustreError: 152920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157097 with bad export cookie 8563081369350136766 [ 3854.271411] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3854.275198] LustreError: 152920:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3854.284480] LustreError: 152250:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 3860.416933] LustreError: 156546:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3860.420234] LustreError: 156546:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3860.493324] Lustre: server umount lustre-MDT0001 complete [ 3864.751424] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3866.019463] Key type lgssc unregistered [ 3866.160441] LNet: 157079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3866.164494] LNetError: 157079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3866.173398] LNet: Removed LNI 192.168.201.101@tcp [ 3866.525583] Key type .llcrypt unregistered [ 3866.527169] Key type ._llcrypt unregistered [ 3877.011292] Key type ._llcrypt registered [ 3877.012690] Key type .llcrypt registered [ 3877.070382] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3877.530927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3877.542442] alg: No test for adler32 (adler32-zlib) [ 3878.454290] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3878.565822] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3880.191201] Key type lgssc registered [ 3880.796939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3884.141941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3884.151826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.374947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.052828] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3890.735315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3890.925749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3892.642995] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3894.941129] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3896.574622] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3899.393418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3899.557165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.815462] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3904.060283] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3909.803169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3917.279985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.283280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3917.288965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.353862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.355038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3920.359174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.361330] Lustre: Skipped 1 previous similar message [ 3922.226152] LustreError: 160714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3922.275938] Lustre: server umount lustre-OST0000 complete [ 3925.472726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.473663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3925.478136] Lustre: Skipped 1 previous similar message [ 3930.592290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3930.595235] Lustre: Skipped 2 previous similar messages [ 3935.712758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3935.715931] Lustre: Skipped 1 previous similar message [ 3938.271155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3938.312397] LustreError: 160915:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3938.316855] LustreError: 160915:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3938.348245] Lustre: server umount lustre-MDT0000 complete [ 3940.009360] LustreError: 158431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157183 with bad export cookie 15485364220279342350 [ 3940.012972] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.014195] LustreError: 158431:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3940.091405] LustreError: 161117:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3940.094186] LustreError: 161117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3940.164156] Lustre: server umount lustre-MDT0001 complete [ 3944.581868] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 3945.880645] Key type lgssc unregistered [ 3946.042595] LNet: 161649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3946.046499] LNetError: 161649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3946.054395] LNet: Removed LNI 192.168.201.101@tcp [ 3946.455268] Key type .llcrypt unregistered [ 3946.457197] Key type ._llcrypt unregistered [ 3961.418821] Key type ._llcrypt registered [ 3961.420241] Key type .llcrypt registered [ 3961.467433] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 3961.944869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3961.955690] alg: No test for adler32 (adler32-zlib) [ 3962.830031] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 3962.929096] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 3964.527235] Key type lgssc registered [ 3964.979467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3968.273154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3968.283089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.490145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.082510] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3974.444054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3974.590242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3976.093391] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3978.167270] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3979.495799] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3982.314757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3982.476233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.691544] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 3986.847641] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3993.772771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4000.224162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4000.227362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4000.232931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.321699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.321880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.193849] LustreError: 165571:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4006.253520] Lustre: server umount lustre-OST0000 complete [ 4009.440634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.446673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4009.449252] Lustre: Skipped 2 previous similar messages [ 4014.560987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.563741] Lustre: Skipped 1 previous similar message [ 4019.681763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.684982] Lustre: Skipped 1 previous similar message [ 4022.239103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4022.279993] LustreError: 165773:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4022.284520] LustreError: 165773:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4022.317405] Lustre: server umount lustre-MDT0000 complete [ 4023.870304] LustreError: 163289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157267 with bad export cookie 6441380213687967342 [ 4023.873552] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.876113] LustreError: 163289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4023.954047] LustreError: 165975:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4023.957156] LustreError: 165975:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4024.020183] Lustre: server umount lustre-MDT0001 complete [ 4028.184766] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4029.408241] Key type lgssc unregistered [ 4029.545582] LNet: 166507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4029.548894] LNetError: 166507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4029.556531] LNet: Removed LNI 192.168.201.101@tcp [ 4029.895096] Key type .llcrypt unregistered [ 4029.896927] Key type ._llcrypt unregistered [ 4039.916572] Key type ._llcrypt registered [ 4039.917877] Key type .llcrypt registered [ 4039.967127] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4040.332398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4040.341657] alg: No test for adler32 (adler32-zlib) [ 4041.238477] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4041.346996] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4042.943314] Key type lgssc registered [ 4043.398813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4047.863555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4047.872164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.069107] Lustre: *** cfs_fail_loc=135, val=0*** [ 4049.125753] LustreError: 167990:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4049.131858] LustreError: 167990:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.101@tcp: cfg command failed: rc = -2 [ 4049.137387] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4049.137387] [ 4049.143704] LustreError: MGC192.168.201.101@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4049.154171] LustreError: 167949:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4049.158664] LustreError: 167949:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4049.163841] LustreError: 167949:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4049.171619] LustreError: 167949:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4049.232654] Lustre: server umount lustre-MDT0000 complete [ 4049.234476] LustreError: 167949:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4052.091087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.317645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.066994] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4057.774274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4057.955845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4059.570216] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4061.910916] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4063.441201] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4067.296887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.297742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.302021] Lustre: Skipped 1 previous similar message [ 4067.308865] Lustre: Skipped 1 previous similar message [ 4068.321248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4068.324712] Lustre: Skipped 1 previous similar message [ 4070.341858] LustreError: 169706:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4070.344553] LustreError: 169706:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4070.374377] Lustre: server umount lustre-MDT0000 complete [ 4071.823385] LustreError: 169082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157315 with bad export cookie 8467872943935295467 [ 4071.825936] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.828440] LustreError: 169082:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4071.901933] LustreError: 169907:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4071.904310] LustreError: 169907:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4071.960919] Lustre: server umount lustre-MDT0001 complete [ 4075.922328] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4077.098781] Key type lgssc unregistered [ 4077.250651] LNet: 170439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4077.254718] LNetError: 170439:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4077.262520] LNet: Removed LNI 192.168.201.101@tcp [ 4077.617221] Key type .llcrypt unregistered [ 4077.618772] Key type ._llcrypt unregistered [ 4086.920747] Key type ._llcrypt registered [ 4086.922438] Key type .llcrypt registered [ 4086.981865] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4087.115360] Key type .llcrypt unregistered [ 4087.116859] Key type ._llcrypt unregistered [ 4094.748600] Key type ._llcrypt registered [ 4094.749891] Key type .llcrypt registered [ 4094.797499] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4095.150930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4095.174069] alg: No test for adler32 (adler32-zlib) [ 4096.046987] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4096.150090] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4097.743163] Key type lgssc registered [ 4098.178471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4101.461144] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4101.470499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4116.959807] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4122.079610] LustreError: 172521:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.101@tcp: failed processing log, type 4: rc = -110 [ 4148.703121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.938471] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4153.064848] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4155.955547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.202536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4156.231623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4158.051545] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4161.507196] LustreError: 172548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.515889] LustreError: 172548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4161.581326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4161.741636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.194935] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4165.194156] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4166.528935] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4167.633312] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4173.282542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4173.283728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4173.287901] Lustre: Skipped 1 previous similar message [ 4173.291675] Lustre: Skipped 1 previous similar message [ 4178.404029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4178.408114] Lustre: Skipped 1 previous similar message [ 4183.522264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4183.524541] Lustre: Skipped 1 previous similar message [ 4187.103169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4187.129114] LustreError: 175149:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4187.198036] Lustre: server umount lustre-OST0000 complete [ 4192.736183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4192.739416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.745216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4193.763269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.948520] LustreError: 175351:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4194.951128] LustreError: 175351:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4194.987652] Lustre: server umount lustre-MDT0000 complete [ 4196.703049] LustreError: 172528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157440 with bad export cookie 7817376501423738908 [ 4196.705656] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4196.708860] LustreError: 172528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4196.794407] LustreError: 175554:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4196.798501] LustreError: 175554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4196.869162] Lustre: server umount lustre-MDT0001 complete [ 4201.470956] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4202.719488] Key type lgssc unregistered [ 4202.877611] LNet: 176085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4202.881052] LNetError: 176085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4202.888376] LNet: Removed LNI 192.168.201.101@tcp [ 4203.250613] Key type .llcrypt unregistered [ 4203.252632] Key type ._llcrypt unregistered [ 4213.708052] Key type ._llcrypt registered [ 4213.709163] Key type .llcrypt registered [ 4213.757590] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4214.237119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4214.273066] alg: No test for adler32 (adler32-zlib) [ 4215.156614] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4215.256623] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4216.840101] Key type lgssc registered [ 4217.289233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4220.368702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4220.377360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.584615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.249506] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4227.247692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4227.420243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4229.055671] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4231.352322] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4232.869151] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4235.762444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4235.956239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.418194] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4240.599824] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4246.183747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4247.507912] Lustre: Failing over lustre-MDT0000 [ 4247.577147] LustreError: 179752:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4247.606883] Lustre: server umount lustre-MDT0000 complete [ 4248.031871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4248.034643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.041101] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.049051] LustreError: 177456:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4251.382408] LustreError: 179615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.617601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.622682] Lustre: Skipped 1 previous similar message [ 4256.492100] LustreError: 179615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.500224] LustreError: 179615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4261.008052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4261.064715] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4261.162996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.187958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4261.612089] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4262.677257] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4264.512655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4266.472980] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4266.483817] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4266.506356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4269.293829] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4276.709547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.709590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4276.715762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4276.720129] Lustre: Skipped 1 previous similar message [ 4281.781929] LustreError: 181038:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4281.784445] LustreError: 181038:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4281.834692] Lustre: server umount lustre-OST0000 complete [ 4286.944847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.944924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.945687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.954450] Lustre: Skipped 2 previous similar messages [ 4289.542692] LustreError: 181239:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4289.545073] LustreError: 181239:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4289.577996] Lustre: server umount lustre-MDT0000 complete [ 4291.038428] LustreError: 179755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157534 with bad export cookie 4488544033690815889 [ 4291.040578] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.042094] LustreError: 179755:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4291.171325] Lustre: server umount lustre-MDT0001 complete [ 4295.400887] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4296.617229] Key type lgssc unregistered [ 4296.753513] LNet: 181971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4296.756597] LNetError: 181971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4296.766419] LNet: Removed LNI 192.168.201.101@tcp [ 4297.129784] Key type .llcrypt unregistered [ 4297.131211] Key type ._llcrypt unregistered [ 4307.786082] Key type ._llcrypt registered [ 4307.787841] Key type .llcrypt registered [ 4307.830916] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4308.431086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4308.442959] alg: No test for adler32 (adler32-zlib) [ 4309.320351] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4309.429879] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4311.023300] Key type lgssc registered [ 4311.537043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4315.042293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4315.050916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.253456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.789305] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4321.227797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.405273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.892977] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4324.982659] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4326.535299] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4329.645741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4329.867428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.423430] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4334.618110] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4341.165405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4345.793456] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4349.409509] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4357.516815] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4366.817391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.819854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.821099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.823818] Lustre: Skipped 1 previous similar message [ 4366.828184] Lustre: Skipped 1 previous similar message [ 4371.939692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4371.942276] Lustre: Skipped 1 previous similar message [ 4371.956212] LustreError: 185778:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4372.006132] Lustre: server umount lustre-OST0000 complete [ 4377.057535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.064184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.067695] Lustre: Skipped 1 previous similar message [ 4379.730690] LustreError: 185979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4379.733553] LustreError: 185979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4379.769683] Lustre: server umount lustre-MDT0000 complete [ 4381.445565] LustreError: 183322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157624 with bad export cookie 8129587438323435200 [ 4381.448349] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4381.449575] LustreError: 183322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4381.537302] LustreError: 186180:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4381.540263] LustreError: 186180:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4381.611487] Lustre: server umount lustre-MDT0001 complete [ 4386.045873] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4387.404195] Key type lgssc unregistered [ 4387.571762] LNet: 186711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.576342] LNetError: 186711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4387.585583] LNet: Removed LNI 192.168.201.101@tcp [ 4387.963526] Key type .llcrypt unregistered [ 4387.964630] Key type ._llcrypt unregistered [ 4397.944734] Key type ._llcrypt registered [ 4397.946428] Key type .llcrypt registered [ 4397.993464] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4398.425872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4398.442040] alg: No test for adler32 (adler32-zlib) [ 4399.333598] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4399.437421] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4401.031168] Key type lgssc registered [ 4401.469848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4404.717503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4404.728216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.961619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.626450] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4411.144944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4411.370531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4413.021148] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4415.037370] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4416.403813] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4419.263739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4419.450727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4422.031724] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4424.314737] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4430.694992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4436.404919] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4443.334653] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4451.872628] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4460.030837] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4471.780402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.780446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.789411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.792621] Lustre: Skipped 1 previous similar message [ 4475.567892] LustreError: 192230:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4475.612409] Lustre: server umount lustre-OST0000 complete [ 4477.920166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4477.923930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.928557] Lustre: Skipped 1 previous similar message [ 4477.931079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.016442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.017323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.264718] LustreError: 192432:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4483.266926] LustreError: 192432:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4483.302507] Lustre: server umount lustre-MDT0000 complete [ 4484.689326] LustreError: 188062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157728 with bad export cookie 10247664084710588278 [ 4484.694049] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4484.694237] LustreError: 188062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4484.770185] LustreError: 192633:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4484.772319] LustreError: 192633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4484.832875] Lustre: server umount lustre-MDT0001 complete [ 4488.692219] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4489.835674] Key type lgssc unregistered [ 4489.978477] LNet: 193164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4489.981257] LNetError: 193164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4489.989470] LNet: Removed LNI 192.168.201.101@tcp [ 4490.332202] Key type .llcrypt unregistered [ 4490.333823] Key type ._llcrypt unregistered [ 4500.041581] Key type ._llcrypt registered [ 4500.042793] Key type .llcrypt registered [ 4500.090271] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4500.507846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4500.534079] alg: No test for adler32 (adler32-zlib) [ 4501.418341] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4501.523647] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4503.119245] Key type lgssc registered [ 4503.623227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4506.700336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4506.707717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.904254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.481370] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4513.060269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4513.222665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4514.862784] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4517.186066] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.707291] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.487760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4521.662686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.970189] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4526.198445] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4532.967567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4543.458685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.460394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.460971] LustreError: Skipped 1 previous similar message [ 4543.466498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4543.469608] Lustre: Skipped 1 previous similar message [ 4546.477980] LustreError: 196948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4546.514561] Lustre: server umount lustre-OST0000 complete [ 4548.576816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.577643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4548.582476] Lustre: Skipped 2 previous similar messages [ 4548.587027] Lustre: Skipped 1 previous similar message [ 4553.696554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4553.699523] Lustre: Skipped 1 previous similar message [ 4558.816192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4558.819456] Lustre: Skipped 1 previous similar message [ 4562.399158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4562.436283] LustreError: 197150:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4562.438672] LustreError: 197150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4562.465108] Lustre: server umount lustre-MDT0000 complete [ 4562.912258] LustreError: 194535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.917595] LustreError: 194535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4563.786330] LustreError: 194515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157807 with bad export cookie 3192535012712261265 [ 4563.789395] LustreError: 194515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4563.789781] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4563.857929] LustreError: 197354:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4563.861280] LustreError: 197354:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4563.919267] Lustre: server umount lustre-MDT0001 complete [ 4567.930123] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4569.084498] Key type lgssc unregistered [ 4569.238477] LNet: 197886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4569.242236] LNetError: 197886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4569.251565] LNet: Removed LNI 192.168.201.101@tcp [ 4569.603750] Key type .llcrypt unregistered [ 4569.605218] Key type ._llcrypt unregistered [ 4577.774118] Key type ._llcrypt registered [ 4577.776090] Key type .llcrypt registered [ 4577.813916] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4578.130493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4578.155082] alg: No test for adler32 (adler32-zlib) [ 4579.018152] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4579.113313] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4580.703180] Key type lgssc registered [ 4581.154342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4584.119613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4584.126833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.290215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.668646] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4589.818336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4589.992886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4591.453664] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4593.434334] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4594.761791] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4597.136282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4597.300604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4599.384694] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4601.429420] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4610.594845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4620.769587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.771160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4620.774550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.776837] LustreError: Skipped 1 previous similar message [ 4620.781380] Lustre: Skipped 1 previous similar message [ 4621.743143] LustreError: 201539:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4621.788923] Lustre: server umount lustre-OST0000 complete [ 4625.888578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.888641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.889246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.892449] Lustre: Skipped 2 previous similar messages [ 4629.372868] LustreError: 201741:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4629.375295] LustreError: 201741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4629.404343] Lustre: server umount lustre-MDT0000 complete [ 4630.629321] LustreError: 199099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763157873 with bad export cookie 16943715670482587803 [ 4630.632488] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4630.633614] LustreError: 199099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4630.706264] LustreError: 201943:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4630.709026] LustreError: 201943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4630.757497] Lustre: server umount lustre-MDT0001 complete [ 4634.497787] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4635.545118] Key type lgssc unregistered [ 4635.675523] LNet: 202473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4635.680089] LNetError: 202473:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4635.687644] LNet: Removed LNI 192.168.201.101@tcp [ 4636.039700] Key type .llcrypt unregistered [ 4636.041060] Key type ._llcrypt unregistered [ 4643.941057] Key type ._llcrypt registered [ 4643.942138] Key type .llcrypt registered [ 4643.980878] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4644.314599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4644.321873] alg: No test for adler32 (adler32-zlib) [ 4645.194766] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4645.286840] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4646.871126] Key type lgssc registered [ 4647.240785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4649.890761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4649.898180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.055350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.287086] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4655.036147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4655.160483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4656.431843] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4658.226272] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4659.470272] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4661.936792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4662.094742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.142923] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4665.988793] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4676.457046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4684.071379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4684.098549] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4684.188189] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4686.285898] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4688.387956] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4695.108974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 4705.674622] Lustre: Permanently deactivating lustre-OST0001 [ 4705.681866] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4714.484873] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4720.924647] Lustre: Permanently reactivating lustre-OST0001 [ 4720.933323] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4720.936480] Lustre: Skipped 2 previous similar messages [ 4723.459157] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.466688] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4723.471352] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4723.476138] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4723.490693] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4733.415049] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.416340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4733.421785] Lustre: Skipped 2 previous similar messages [ 4736.366127] LustreError: 207207:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4736.385739] Lustre: server umount lustre-OST0001 complete [ 4738.015883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4738.018281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.023878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4738.026795] Lustre: Skipped 1 previous similar message [ 4738.532883] LustreError: 205297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.539444] LustreError: 205297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4743.650515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.651244] LustreError: 205294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.653096] Lustre: Skipped 2 previous similar messages [ 4743.659248] LustreError: 205294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4748.770930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4748.771512] LustreError: 205297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.773659] Lustre: Skipped 1 previous similar message [ 4748.781075] LustreError: 205297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4751.839140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4751.860174] LustreError: 207408:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4751.863084] LustreError: 207408:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4751.907374] Lustre: server umount lustre-OST0000 complete [ 4753.888961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4753.890145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4753.893201] Lustre: Skipped 2 previous similar messages [ 4759.493604] LustreError: 207612:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4759.496422] LustreError: 207612:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4759.533658] Lustre: server umount lustre-MDT0000 complete [ 4761.173536] LustreError: 206226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158004 with bad export cookie 1289760631795932703 [ 4761.176046] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.178237] LustreError: 206226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4761.323200] Lustre: server umount lustre-MDT0001 complete [ 4765.229208] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 4766.404061] Key type lgssc unregistered [ 4766.555438] LNet: 208346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4766.557895] LNetError: 208346:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4766.569416] LNet: Removed LNI 192.168.201.101@tcp [ 4766.917339] Key type .llcrypt unregistered [ 4766.918380] Key type ._llcrypt unregistered [ 4779.067692] Key type ._llcrypt registered [ 4779.069365] Key type .llcrypt registered [ 4779.113526] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 4785.064350] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 4785.587760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4785.636604] alg: No test for adler32 (adler32-zlib) [ 4786.494782] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 4786.587360] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 4788.167186] Key type lgssc registered [ 4788.579903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4791.575713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4793.778971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4795.862955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4798.061375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.487566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4800.504837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4800.510366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.598165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4801.612340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4801.645125] Lustre: lustre-MDT0000: new disk, initializing [ 4801.666835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.673351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4802.915769] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4807.006239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4807.039826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4807.076728] Lustre: 212263:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4807.102930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4807.105928] Lustre: Skipped 1 previous similar message [ 4807.148269] Lustre: lustre-MDT0001: new disk, initializing [ 4807.195155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4807.210918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4807.214558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4808.598251] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4811.742582] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.966847] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4815.283649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4815.307987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4815.397251] Lustre: lustre-OST0000: new disk, initializing [ 4815.400421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4815.431645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.343851] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4818.745041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4818.748034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4818.783933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4820.509914] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4821.901755] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4821.981588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4823.363843] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4823.444533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4829.156442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.156518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.160383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.164108] Lustre: Skipped 1 previous similar message [ 4830.127120] LustreError: 215204:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4830.169594] Lustre: server umount lustre-OST0000 complete [ 4832.735990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4832.738820] LustreError: Skipped 1 previous similar message [ 4832.739842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.743296] Lustre: Skipped 1 previous similar message [ 4832.745130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.272128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.272705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.498740] LustreError: 215405:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4837.500675] LustreError: 215405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4837.525817] Lustre: server umount lustre-MDT0000 complete [ 4838.723946] LustreError: 213045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158082 with bad export cookie 8527574281119656779 [ 4838.726442] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.726816] LustreError: 213045:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4838.792167] LustreError: 215606:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4838.793752] LustreError: 215606:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4838.840458] Lustre: server umount lustre-MDT0001 complete [ 4841.477856] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 17:08:04 (1763158084) [ 4843.623661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4843.773092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.958640] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4847.694838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4847.797680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4848.929217] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4850.791646] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4852.020795] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4854.361259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4856.506838] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 4858.416065] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4869.450648] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4872.888064] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4879.414150] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4886.977950] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4905.380271] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4905.382685] Lustre: Skipped 1 previous similar message [ 4922.537476] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4922.540505] Lustre: Skipped 1 previous similar message [ 4956.043444] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4956.046995] Lustre: Skipped 3 previous similar messages [ 5025.786616] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5025.789540] Lustre: Skipped 7 previous similar messages [ 5048.291490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5048.291807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.293965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5048.294058] Lustre: Skipped 1 previous similar message [ 5048.295168] LustreError: Skipped 1 previous similar message [ 5048.299940] Lustre: Skipped 1 previous similar message [ 5052.975096] LustreError: 219792:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5052.976950] LustreError: 219792:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5053.012333] Lustre: server umount lustre-OST0000 complete [ 5058.016716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5058.020428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.027540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5058.030462] Lustre: Skipped 1 previous similar message [ 5060.479547] LustreError: 219997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5060.482274] LustreError: 219997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5060.511554] Lustre: server umount lustre-MDT0000 complete [ 5061.692257] LustreError: 216256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158305 with bad export cookie 8527574281119657647 [ 5061.695742] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5061.696445] LustreError: 216256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5061.811106] Lustre: server umount lustre-MDT0001 complete [ 5065.224704] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5066.211256] Key type lgssc unregistered [ 5066.336323] LNet: 220730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5066.338686] LNetError: 220730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5066.345237] LNet: Removed LNI 192.168.201.101@tcp [ 5066.617697] Key type .llcrypt unregistered [ 5066.619255] Key type ._llcrypt unregistered [ 5075.887620] Key type ._llcrypt registered [ 5075.888615] Key type .llcrypt registered [ 5075.930387] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5076.367474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5076.406341] alg: No test for adler32 (adler32-zlib) [ 5077.269429] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5077.361384] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5078.943173] Key type lgssc registered [ 5079.400067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5082.508427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5082.516419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.706460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5085.102465] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5088.170838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5088.331982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5089.704983] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5091.530200] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5092.780478] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5095.132556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5095.264099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5097.256130] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_default_debug -1 all [ 5099.421670] Lustre: DEBUG MARKER: oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5114.849278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.850076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.850749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.853428] Lustre: Skipped 1 previous similar message [ 5119.971328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.973857] Lustre: Skipped 2 previous similar messages [ 5125.091064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.092868] Lustre: Skipped 1 previous similar message [ 5129.183161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5129.201085] LustreError: 224518:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5129.241397] Lustre: server umount lustre-OST0000 complete [ 5134.303820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5134.306821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.311698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5135.328733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.637901] LustreError: 224724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5136.640343] LustreError: 224724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5136.671886] Lustre: server umount lustre-MDT0000 complete [ 5138.033353] LustreError: 222080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763158381 with bad export cookie 290173124853084881 [ 5138.037536] LustreError: MGC192.168.201.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.037745] LustreError: 222080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5138.116989] LustreError: 224927:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5138.119719] LustreError: 224927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5138.174801] Lustre: server umount lustre-MDT0001 complete [ 5141.643356] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5142.704567] Key type lgssc unregistered [ 5142.825431] LNet: 225457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5142.828981] LNetError: 225457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5142.835303] LNet: Removed LNI 192.168.201.101@tcp [ 5143.140389] Key type .llcrypt unregistered [ 5143.141436] Key type ._llcrypt unregistered [ 5149.878214] Key type ._llcrypt registered [ 5149.879046] Key type .llcrypt registered [ 5149.912542] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5150.011299] Key type .llcrypt unregistered [ 5150.012807] Key type ._llcrypt unregistered [ 5157.917853] Key type ._llcrypt registered [ 5157.919314] Key type .llcrypt registered [ 5157.964889] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing set_hostid [ 5164.455774] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing load_modules_local [ 5164.783296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5164.825237] alg: No test for adler32 (adler32-zlib) [ 5165.694575] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 5165.779195] LNet: Added LNI 192.168.201.101@tcp [8/256/0/180] [ 5167.367136] Key type lgssc registered [ 5167.798654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5170.614015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5172.793704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5174.813931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5176.977971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5177.547182] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4985 sec ========== 17:13:40 (1763158420) [ 5178.115520] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:13:41 (1763158421) === [ 5179.257469] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:13:42 (1763158422) === [ 5192.124692] Lustre: DEBUG MARKER: oleg101-server.virtnet: executing unload_modules_local [ 5193.218649] Key type lgssc unregistered [ 5193.360484] LNet: 231535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5193.363666] LNetError: 231535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5193.372446] LNet: Removed LNI 192.168.201.101@tcp [ 5193.670312] Key type .llcrypt unregistered [ 5193.671202] Key type ._llcrypt unregistered