[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 644176493 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003247] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007754] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.014041] *** VALIDATE selinux *** [ 0.022610] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026685] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030117] *** VALIDATE tmpfs *** [ 0.032014] *** VALIDATE proc *** [ 0.033279] *** VALIDATE cgroup *** [ 0.034018] *** VALIDATE cgroup2 *** [ 0.035244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038025] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042255] debug: unmapping init [mem 0xffffffff9ba59000-0xffffffff9ba60fff] [ 0.044178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051026] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053260] rcu: Hierarchical SRCU implementation. [ 0.055360] smp: Bringing up secondary CPUs ... [ 0.056520] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.061019] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.191000] node 0 deferred pages initialised in 124ms [ 0.195394] devtmpfs: initialized [ 0.196205] x86/mm: Memory block size: 128MB [ 0.199054] gcov: version magic: 0x41383552 [ 0.201487] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.205118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.208413] pinctrl core: initialized pinctrl subsystem [ 0.210229] [ 0.210712] ************************************************************* [ 0.213018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.215016] ** ** [ 0.217018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.219017] ** ** [ 0.221016] ** This means that this kernel is built to expose internal ** [ 0.224016] ** IOMMU data structures, which may compromise security on ** [ 0.226017] ** your system. ** [ 0.228017] ** ** [ 0.231021] ** If you see this message and you are not debugging the ** [ 0.233015] ** kernel, report this immediately to your vendor! ** [ 0.236025] ** ** [ 0.238017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240017] ************************************************************* [ 0.242747] NET: Registered protocol family 16 [ 0.245491] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.247069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.250086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253120] cpuidle: using governor menu [ 0.254689] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257729] PCI: Using configuration type 1 for base access [ 0.260133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.270087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.278166] cryptd: max_cpu_qlen set to 1000 [ 0.281225] ACPI: Added _OSI(Module Device) [ 0.282014] ACPI: Added _OSI(Processor Device) [ 0.283014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289209] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.295362] ACPI: Interpreter enabled [ 0.296078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298016] ACPI: Using IOAPIC for interrupt routing [ 0.300142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.303465] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313710] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.316058] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.318022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.320122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.325392] acpiphp: Slot [2] registered [ 0.326225] acpiphp: Slot [3] registered [ 0.328137] acpiphp: Slot [4] registered [ 0.329115] acpiphp: Slot [5] registered [ 0.331225] acpiphp: Slot [6] registered [ 0.332180] acpiphp: Slot [7] registered [ 0.334128] acpiphp: Slot [8] registered [ 0.335144] acpiphp: Slot [9] registered [ 0.337147] acpiphp: Slot [10] registered [ 0.338119] acpiphp: Slot [11] registered [ 0.339113] acpiphp: Slot [12] registered [ 0.341122] acpiphp: Slot [13] registered [ 0.343151] acpiphp: Slot [14] registered [ 0.345133] acpiphp: Slot [15] registered [ 0.346129] acpiphp: Slot [16] registered [ 0.348106] acpiphp: Slot [17] registered [ 0.349116] acpiphp: Slot [18] registered [ 0.351168] acpiphp: Slot [19] registered [ 0.352136] acpiphp: Slot [20] registered [ 0.354112] acpiphp: Slot [21] registered [ 0.355115] acpiphp: Slot [22] registered [ 0.356098] acpiphp: Slot [23] registered [ 0.358133] acpiphp: Slot [24] registered [ 0.359121] acpiphp: Slot [25] registered [ 0.361165] acpiphp: Slot [26] registered [ 0.362129] acpiphp: Slot [27] registered [ 0.364146] acpiphp: Slot [28] registered [ 0.365126] acpiphp: Slot [29] registered [ 0.367130] acpiphp: Slot [30] registered [ 0.368134] acpiphp: Slot [31] registered [ 0.370102] PCI host bridge to bus 0000:00 [ 0.371025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.374041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.376032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.378034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.380037] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.383058] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390524] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.395683] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.404019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.409062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.412026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.415036] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.417025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.419573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.423756] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.431766] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.438022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.453024] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.460955] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.465779] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.474028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.495021] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.504109] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.518024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.535022] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.545349] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.577018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.591000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.600016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.608015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.624016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.634586] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.639019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.644023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654000] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.664760] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.670023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.675021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.690029] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.700973] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.705395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.707389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.710388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.712229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.717147] iommu: Default domain type: Passthrough [ 0.719620] SCSI subsystem initialized [ 0.721146] ACPI: bus type USB registered [ 0.722148] usbcore: registered new interface driver usbfs [ 0.724110] usbcore: registered new interface driver hub [ 0.726112] usbcore: registered new device driver usb [ 0.727236] pps_core: LinuxPPS API ver. 1 registered [ 0.729015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.732118] PTP clock support registered [ 0.734120] EDAC MC: Ver: 3.0.0 [ 0.736166] PCI: Using ACPI for IRQ routing [ 0.738000] NetLabel: Initializing [ 0.739024] NetLabel: domain hash size = 128 [ 0.741022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.743126] NetLabel: unlabeled traffic allowed by default [ 0.748076] vgaarb: loaded [ 0.749405] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757000] clocksource: Switched to clocksource kvm-clock [ 0.881644] VFS: Disk quotas dquot_6.6.0 [ 0.883069] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.885275] *** VALIDATE ramfs *** [ 0.886327] *** VALIDATE hugetlbfs *** [ 0.887683] pnp: PnP ACPI init [ 0.889787] pnp: PnP ACPI: found 6 devices [ 0.907074] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909768] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.911756] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.913854] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915936] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918118] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.920759] NET: Registered protocol family 2 [ 0.923334] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928215] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931925] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.936675] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939812] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942509] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.945469] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948029] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951348] NET: Registered protocol family 1 [ 0.953972] RPC: Registered named UNIX socket transport module. [ 0.956186] RPC: Registered udp transport module. [ 0.957710] RPC: Registered tcp transport module. [ 0.959241] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.962566] NET: Registered protocol family 44 [ 0.963985] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.966167] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.968043] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.970172] PCI: CLS 0 bytes, default 64 [ 0.972261] Unpacking initramfs... [ 2.379305] debug: unmapping init [mem 0xffff8d9afcc54000-0xffff8d9afffbffff] [ 2.383042] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.384928] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.387508] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.883109] Initialise system trusted keyrings [ 2.884678] Key type blacklist registered [ 2.886482] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.895573] zbud: loaded [ 2.898765] *** VALIDATE nfs *** [ 2.900054] *** VALIDATE nfs4 *** [ 2.901712] pstore: using deflate compression [ 2.904786] Platform Keyring initialized [ 3.009757] NET: Registered protocol family 38 [ 3.011685] Key type asymmetric registered [ 3.013272] Asymmetric key parser 'x509' registered [ 3.014996] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.017837] io scheduler mq-deadline registered [ 3.019353] io scheduler kyber registered [ 3.021171] io scheduler bfq registered [ 3.023944] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.027123] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.030187] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.033214] ACPI: Power Button [PWRF] [ 3.127295] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.219383] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.401425] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.493228] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.677912] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.706625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.736494] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.741416] Non-volatile memory driver v1.3 [ 3.742849] Linux agpgart interface v0.103 [ 3.776163] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.779954] vda: detected capacity change from 0 to 68521984 [ 3.794522] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.797406] vdb: detected capacity change from 0 to 1073741824 [ 3.814431] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.817472] vdc: detected capacity change from 0 to 2621440000 [ 3.833644] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.836730] vdd: detected capacity change from 0 to 2621440000 [ 3.852369] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.854302] vde: detected capacity change from 0 to 4294967296 [ 3.872566] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.875658] vdf: detected capacity change from 0 to 4294967296 [ 3.884182] libphy: Fixed MDIO Bus: probed [ 3.889590] usbcore: registered new interface driver usbserial_generic [ 3.891923] usbserial: USB Serial support registered for generic [ 3.894208] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.898393] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.900327] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.902864] mousedev: PS/2 mouse device common for all mice [ 3.905874] rtc_cmos 00:05: RTC can wake from S4 [ 3.908221] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.912554] rtc_cmos 00:05: registered as rtc0 [ 3.914623] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.917267] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.920997] intel_pstate: CPU model not supported [ 3.931401] hid: raw HID events driver (C) Jiri Kosina [ 3.933573] usbcore: registered new interface driver usbhid [ 3.938039] usbhid: USB HID core driver [ 3.938310] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.939307] drop_monitor: Initializing network drop monitor service [ 3.944322] Initializing XFRM netlink socket [ 3.946497] NET: Registered protocol family 10 [ 3.949828] Segment Routing with IPv6 [ 3.951815] NET: Registered protocol family 17 [ 3.954095] mpls_gso: MPLS GSO support [ 3.961574] RAS: Correctable Errors collector initialized. [ 3.963855] AVX version of gcm_enc/dec engaged. [ 3.965701] AES CTR mode by8 optimization enabled [ 4.051426] sched_clock: Marking stable (4051402700, 0)->(5006742683, -955339983) [ 4.055907] registered taskstats version 1 [ 4.058570] Loading compiled-in X.509 certificates [ 4.060792] zswap: loaded using pool lzo/zbud [ 4.088831] Key type big_key registered [ 4.105537] Key type encrypted registered [ 4.107698] ima: No TPM chip found, activating TPM-bypass! [ 4.110377] ima: Allocated hash algorithm: sha1 [ 4.112422] ima: No architecture policies found [ 4.114623] evm: Initialising EVM extended attributes: [ 4.116952] evm: security.selinux [ 4.118103] evm: security.ima [ 4.119275] evm: security.capability [ 4.121659] evm: HMAC attrs: 0x1 [ 4.125461] rtc_cmos 00:05: setting system clock to 2025-11-16 22:46:41 UTC (1763333201) [ 4.131736] debug: unmapping init [mem 0xffffffff9ca03000-0xffffffff9cbfffff] [ 4.134754] debug: unmapping init [mem 0xffffffff9b782000-0xffffffff9ba58fff] [ 4.144252] Write protecting the kernel read-only data: 28672k [ 4.148884] debug: unmapping init [mem 0xffffffff99e03000-0xffffffff99ffffff] [ 4.151820] debug: unmapping init [mem 0xffffffff9a714000-0xffffffff9a7fffff] [ 4.186081] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.194648] systemd[1]: Detected virtualization kvm. [ 4.196725] systemd[1]: Detected architecture x86-64. [ 4.198610] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.227724] systemd[1]: No hostname configured. [ 4.229318] systemd[1]: Set hostname to . [ 4.231236] random: systemd: uninitialized urandom read (16 bytes read) [ 4.233527] systemd[1]: Initializing machine ID from random generator. [ 4.276869] random: ln: uninitialized urandom read (6 bytes read) [ 4.374463] random: systemd: uninitialized urandom read (16 bytes read) [ 4.377240] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.383546] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.389709] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.111825] device-mapper: uevent: version 1.0.3 [ 5.114173] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.797642] virtio_net virtio0 ens2: renamed from eth0 [ 5.830737] scsi host0: ata_piix [ 5.876243] scsi host1: ata_piix [ 5.878560] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.883623] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.900898] random: fast init done [ 10.425459] dracut-initqueue[592]: RTNETLINK answers: File exists [ 10.628136] random: crng init done [ 10.629430] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.216667] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.461076] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.773164] SELinux: Disabled at runtime. [ 12.839961] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.849745] systemd[1]: Detected virtualization kvm. [ 12.851809] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.474827] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.479443] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.485190] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.490796] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.494484] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.505911] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.512337] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 13.704069] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 14.001068] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.374694] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.398837] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.564764] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.576533] EDAC sbridge: Ver: 1.1.2 [ 16.433560] Key type dns_resolver registered [ 16.741759] NFS: Registering the id_resolver key type [ 16.743570] Key type id_resolver registered [ 16.744845] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 49.300631] libcfs: loading out-of-tree module taints kernel. [ 49.329790] Key type ._llcrypt registered [ 49.331268] Key type .llcrypt registered [ 49.398983] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 63.312741] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 64.401765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.413829] alg: No test for adler32 (adler32-zlib) [ 65.644597] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 66.210956] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 67.927181] Key type lgssc registered [ 69.166711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.218155] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 81.788322] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.986715] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.326449] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.561560] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.723497] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 121.557714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.622870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 121.643484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 122.946377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 122.982564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 123.120087] Lustre: lustre-MDT0000: new disk, initializing [ 123.204662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.227712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 126.765306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 138.805276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.860501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 138.907917] Lustre: 6517:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 138.944783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 138.947597] Lustre: Skipped 1 previous similar message [ 139.038929] Lustre: lustre-MDT0001: new disk, initializing [ 139.102327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.131597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 139.146140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 142.179623] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.916163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 153.224686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.267581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 153.427911] Lustre: lustre-OST0000: new disk, initializing [ 153.434476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 153.501445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.995578] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.752799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 158.760513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 158.850156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 171.175629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.231802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.329880] Lustre: lustre-OST0001: new disk, initializing [ 171.337400] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 171.407236] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.941797] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.767519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 180.772698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 180.873918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 186.556369] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 193.254652] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 204.090317] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 208.888539] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 213.123162] Lustre: DEBUG MARKER: Client: 2.16.61.52 [ 215.924947] Lustre: DEBUG MARKER: MDS: 2.16.61.52 [ 218.429930] Lustre: DEBUG MARKER: OSS: 2.16.61.52 [ 220.295410] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:50:15 EST 2025 [ 224.026184] hrtimer: interrupt took 14106619 ns [ 239.328456] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 240.818870] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 256.995992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 257.009214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.040153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 257.519744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.530296] Lustre: Skipped 2 previous similar messages [ 262.640573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.646176] Lustre: Skipped 7 previous similar messages [ 262.715405] LustreError: 12223:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 262.863585] Lustre: server umount lustre-MDT0000 complete [ 266.512389] LustreError: 9485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333463 with bad export cookie 1440991273679189169 [ 266.512857] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.520385] LustreError: 9485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 267.744271] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.747833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 267.750269] Lustre: Skipped 1 previous similar message [ 267.766320] Lustre: Skipped 1 previous similar message [ 271.776194] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 272.738326] LustreError: 12424:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 272.884185] LustreError: 12424:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 273.140077] Lustre: server umount lustre-MDT0001 complete [ 277.269948] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 277.273117] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 277.327228] Lustre: server umount lustre-OST0000 complete [ 280.936996] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 280.967161] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 281.337946] Lustre: server umount lustre-OST0001 complete [ 287.708700] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 295.721580] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 304.768318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 310.470338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 316.547409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.106235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.587286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.654907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 328.866431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 328.916769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 329.018359] Lustre: lustre-MDT0000: new disk, initializing [ 329.101837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.118463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 332.806870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 343.085885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.141418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 343.255557] Lustre: 15010:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 343.329650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 343.335255] Lustre: Skipped 1 previous similar message [ 343.381889] Lustre: lustre-MDT0001: new disk, initializing [ 343.533617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 343.599345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 343.606589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 347.238211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 353.363807] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 356.222549] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 361.263946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.323341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 361.512632] Lustre: lustre-OST0000: new disk, initializing [ 361.518566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 361.601398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.942550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 362.966927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 363.051842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 365.996945] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 374.198122] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 378.877162] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 379.180717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 383.414339] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 383.697362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 388.600237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 388.612472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.648598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 388.651421] Lustre: Skipped 1 previous similar message [ 392.536280] LustreError: 17955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 392.543691] LustreError: 17955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 392.734041] Lustre: server umount lustre-OST0000 complete [ 398.821509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.829249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 398.831177] Lustre: Skipped 2 previous similar messages [ 402.885258] LustreError: 18156:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 402.888778] LustreError: 18156:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 403.041045] Lustre: server umount lustre-MDT0000 complete [ 403.942297] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.952099] LustreError: 15021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 407.204481] LustreError: 16762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333604 with bad export cookie 1440991273679190702 [ 407.211493] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.235340] LustreError: 16762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 407.640295] Lustre: server umount lustre-MDT0001 complete [ 411.451359] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:53:26 (1763333606) [ 417.334257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.661261] LustreError: 18883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 417.735884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 421.010534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 422.886144] LustreError: 18883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.002485] LustreError: 18883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.131865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 431.368923] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 435.268552] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 438.716894] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 445.288146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 450.659107] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 454.603568] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 473.571435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.580517] LustreError: Skipped 1 previous similar message [ 473.582195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.585974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.592779] Lustre: Skipped 1 previous similar message [ 473.597112] Lustre: Skipped 1 previous similar message [ 484.319439] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 484.355363] LustreError: 21150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 484.359742] LustreError: 21150:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 484.437689] Lustre: server umount lustre-OST0000 complete [ 501.215901] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 501.340684] Lustre: server umount lustre-MDT0000 complete [ 504.296685] LustreError: 19595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.307566] LustreError: 19595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 504.840499] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333702 with bad export cookie 1440991273679191584 [ 504.841728] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.853634] LustreError: 18863:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 505.272292] Lustre: server umount lustre-MDT0001 complete [ 514.541492] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 516.855290] Key type lgssc unregistered [ 517.173600] LNet: 22090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.191470] LNetError: 22090:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.212590] LNet: Removed LNI 192.168.202.133@tcp [ 518.029286] Key type .llcrypt unregistered [ 518.035603] Key type ._llcrypt unregistered [ 536.817870] Key type ._llcrypt registered [ 536.819319] Key type .llcrypt registered [ 536.884071] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 537.629611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 537.721689] alg: No test for adler32 (adler32-zlib) [ 538.738306] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 538.943452] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 540.679168] Key type lgssc registered [ 542.216607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 550.782592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 550.797453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.353526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.580603] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 561.634956] LustreError: 23464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.650942] LustreError: 23464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 565.449476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 565.793525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 568.987671] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 574.075448] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 577.417752] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 583.492688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 583.844483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.230611] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 594.534280] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 597.177512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 611.809521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.823763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.841430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 612.835798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 612.836530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.839084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 614.743927] LustreError: 26152:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 614.942243] Lustre: server umount lustre-OST0000 complete [ 622.052349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.055296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.072466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.075557] Lustre: Skipped 1 previous similar message [ 628.194642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.197169] Lustre: Skipped 3 previous similar messages [ 633.312592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 633.314237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.322642] Lustre: Skipped 1 previous similar message [ 633.393291] LustreError: 26354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 633.403436] LustreError: 26354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 633.463093] Lustre: server umount lustre-MDT0000 complete [ 636.526799] LustreError: 23443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333833 with bad export cookie 1062527601724073853 [ 636.528725] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.537758] LustreError: 23443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 636.679638] LustreError: 26557:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 636.686335] LustreError: 26557:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 636.834714] Lustre: server umount lustre-MDT0001 complete [ 644.971911] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 647.187171] Key type lgssc unregistered [ 647.426953] LNet: 27088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.438051] LNetError: 27088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.448672] LNet: Removed LNI 192.168.202.133@tcp [ 648.134536] Key type .llcrypt unregistered [ 648.135682] Key type ._llcrypt unregistered [ 663.948513] Key type ._llcrypt registered [ 663.950318] Key type .llcrypt registered [ 664.043162] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 664.787632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.799141] alg: No test for adler32 (adler32-zlib) [ 665.858128] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 666.119376] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 667.751159] Key type lgssc registered [ 668.813746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.698110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 675.734311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.173701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.383768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 687.585428] LustreError: 28465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.592144] LustreError: 28465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 687.604722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 688.008543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 691.615324] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 696.591288] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 699.601134] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 709.778331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 710.010955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.711182] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 719.721311] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 722.297507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 732.645398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 732.647343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 732.657864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 732.668339] Lustre: Skipped 1 previous similar message [ 737.763714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.885665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.888695] Lustre: Skipped 1 previous similar message [ 743.391155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 743.445122] LustreError: 31054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 743.601834] Lustre: server umount lustre-OST0000 complete [ 748.005671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.009463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.030551] Lustre: Skipped 2 previous similar messages [ 748.039546] Lustre: Skipped 1 previous similar message [ 753.121961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 753.128119] Lustre: Skipped 2 previous similar messages [ 753.729079] LustreError: 31257:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 753.738628] LustreError: 31257:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 753.841464] Lustre: server umount lustre-MDT0000 complete [ 757.205489] LustreError: 29185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333954 with bad export cookie 15265465186618103193 [ 757.208140] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 757.223365] LustreError: 29185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 757.367320] LustreError: 31458:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 757.375389] LustreError: 31458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 757.634235] Lustre: server umount lustre-MDT0001 complete [ 766.765518] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 769.132230] Key type lgssc unregistered [ 769.435554] LNet: 31990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.444020] LNetError: 31990:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.458194] LNet: Removed LNI 192.168.202.133@tcp [ 770.140172] Key type .llcrypt unregistered [ 770.141626] Key type ._llcrypt unregistered [ 788.683220] Key type ._llcrypt registered [ 788.684868] Key type .llcrypt registered [ 788.849478] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 789.778389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 789.789468] alg: No test for adler32 (adler32-zlib) [ 790.808550] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 791.026715] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 792.679810] Key type lgssc registered [ 793.664712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.324898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.356597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.823774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.497598] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 812.002946] LustreError: 33363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 812.021872] LustreError: 33363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 813.411520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 813.694917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 816.755301] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 821.821652] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 825.116481] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 830.713217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.076931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.180950] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 841.241328] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 847.473428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 858.084875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.085227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.092060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.093834] Lustre: Skipped 1 previous similar message [ 863.204646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 863.212242] Lustre: Skipped 2 previous similar messages [ 863.952602] LustreError: 35640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 864.214565] Lustre: server umount lustre-OST0000 complete [ 868.325838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.326584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.335396] Lustre: Skipped 1 previous similar message [ 868.345528] Lustre: Skipped 1 previous similar message [ 873.449875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.454236] Lustre: Skipped 1 previous similar message [ 874.444215] LustreError: 35855:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 874.447631] LustreError: 35855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 874.646515] Lustre: server umount lustre-MDT0000 complete [ 878.261952] LustreError: 33342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334075 with bad export cookie 13544716932576362422 [ 878.266892] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.271486] LustreError: 33342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 878.405417] LustreError: 36056:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 878.413075] LustreError: 36056:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 878.615239] Lustre: server umount lustre-MDT0001 complete [ 888.899837] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 891.243296] Key type lgssc unregistered [ 891.505276] LNet: 36588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.512414] LNetError: 36588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.535899] LNet: Removed LNI 192.168.202.133@tcp [ 892.422823] Key type .llcrypt unregistered [ 892.425907] Key type ._llcrypt unregistered [ 914.511128] Key type ._llcrypt registered [ 914.512739] Key type .llcrypt registered [ 914.605425] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 915.497861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.507250] alg: No test for adler32 (adler32-zlib) [ 916.452364] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 916.644610] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 918.287175] Key type lgssc registered [ 919.244202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.578761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.595260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.964152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.803321] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 937.061889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 937.364723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.328845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 944.788922] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 947.574908] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 953.777033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 954.157871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.215926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 964.885514] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 973.639708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 980.695746] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 983.009745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.017922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.028206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.035768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.037238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.042467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.663861] LustreError: 40251:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 985.744946] Lustre: server umount lustre-OST0000 complete [ 1003.201574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.204576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.209622] Lustre: Skipped 1 previous similar message [ 1003.217430] Lustre: Skipped 2 previous similar messages [ 1008.608535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.616821] Lustre: Skipped 1 previous similar message [ 1009.397500] LustreError: 40627:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1009.401996] LustreError: 40627:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1009.520621] Lustre: server umount lustre-MDT0000 complete [ 1009.634243] LustreError: 37983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.662291] LustreError: 37983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1012.508872] LustreError: 37966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334209 with bad export cookie 17923644080195549443 [ 1012.514131] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1012.522539] LustreError: 37966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1018.733165] LustreError: 40829:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1018.738154] LustreError: 40829:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1018.924491] Lustre: server umount lustre-MDT0001 complete [ 1028.158043] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1030.274928] Key type lgssc unregistered [ 1030.505760] LNet: 41362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.511168] LNetError: 41362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.538939] LNet: Removed LNI 192.168.202.133@tcp [ 1031.080356] Key type .llcrypt unregistered [ 1031.082642] Key type ._llcrypt unregistered [ 1047.900587] Key type ._llcrypt registered [ 1047.902499] Key type .llcrypt registered [ 1048.015891] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1048.748382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.776469] alg: No test for adler32 (adler32-zlib) [ 1049.751556] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1049.917595] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1051.567184] Key type lgssc registered [ 1052.391943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1059.161645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.186626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.535621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.351323] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1070.051102] LustreError: 42735:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.079328] LustreError: 42735:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1073.123435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1073.368168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1076.192992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1081.707964] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1086.013117] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1092.969476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1093.322978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.781416] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1105.148608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1105.656344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1111.331296] LustreError: 44981:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1111.475876] Lustre: server umount lustre-MDT0000 complete [ 1112.316359] LustreError: 42731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.459175] LustreError: 42716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334311 with bad export cookie 14154298945809068791 [ 1114.465902] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.470375] LustreError: 42716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1116.131708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.138073] LustreError: 42731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.140783] Lustre: Skipped 1 previous similar message [ 1116.172274] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1117.406105] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [ 1119.140934] LustreError: 42731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.142783] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1119.148392] LustreError: 42731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1120.632423] LustreError: 45182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1120.637706] LustreError: 45182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1120.784416] Lustre: server umount lustre-MDT0001 complete [ 1141.058508] LustreError: 45405:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1141.068455] LustreError: 45405:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1141.205950] Lustre: server umount lustre-OST0000 complete [ 1156.223513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1158.751267] Key type lgssc unregistered [ 1159.041641] LNet: 46243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.052700] LNetError: 46243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.071178] LNet: Removed LNI 192.168.202.133@tcp [ 1159.868296] Key type .llcrypt unregistered [ 1159.869892] Key type ._llcrypt unregistered [ 1180.559439] Key type ._llcrypt registered [ 1180.560910] Key type .llcrypt registered [ 1180.655342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1181.588914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.617327] alg: No test for adler32 (adler32-zlib) [ 1182.632289] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1182.798581] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1184.447228] Key type lgssc registered [ 1185.510975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.341614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.354861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1207.839290] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1212.959216] LustreError: 47608:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1239.586437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.253186] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1249.238959] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1415.367368] LustreError: 48325:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1415.483987] Lustre: server umount lustre-OST0000 complete [ 1429.808147] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1432.335559] Key type lgssc unregistered [ 1432.586720] LNet: 49163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.597623] LNetError: 49163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.612742] LNet: Removed LNI 192.168.202.133@tcp [ 1433.290436] Key type .llcrypt unregistered [ 1433.297800] Key type ._llcrypt unregistered [ 1451.312331] Key type ._llcrypt registered [ 1451.313979] Key type .llcrypt registered [ 1451.443543] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1452.071555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.156800] alg: No test for adler32 (adler32-zlib) [ 1453.156734] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1453.427461] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1455.087200] Key type lgssc registered [ 1456.209357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1463.383930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1463.417498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.942804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.170537] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1475.553201] LustreError: 50539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.581326] LustreError: 50539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1476.330935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1476.660868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.213992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1485.369856] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1488.398381] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1493.522331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1493.779869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.363172] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1501.412383] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1506.278491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1506.279841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.299244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.304547] LustreError: 52780:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1509.385263] Lustre: server umount lustre-OST0000 complete [ 1512.419227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1512.429123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.442923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.488830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.491486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.516470] Lustre: Skipped 1 previous similar message [ 1518.362824] LustreError: 52982:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1518.366244] LustreError: 52982:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1518.441284] Lustre: server umount lustre-MDT0000 complete [ 1520.611164] LustreError: 52593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.622543] LustreError: 52593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1521.099989] LustreError: 50518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334718 with bad export cookie 3757772998305058788 [ 1521.106449] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.116956] LustreError: 50518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1521.239736] LustreError: 53184:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1521.242634] LustreError: 53184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1521.368091] Lustre: server umount lustre-MDT0001 complete [ 1527.807560] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1529.820181] Key type lgssc unregistered [ 1530.089911] LNet: 53714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.097756] LNetError: 53714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.112864] LNet: Removed LNI 192.168.202.133@tcp [ 1531.786366] Key type .llcrypt unregistered [ 1531.788165] Key type ._llcrypt unregistered [ 1547.027740] Key type ._llcrypt registered [ 1547.029621] Key type .llcrypt registered [ 1547.104732] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1547.838837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.862060] alg: No test for adler32 (adler32-zlib) [ 1548.863818] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1549.010991] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1550.623322] Key type lgssc registered [ 1551.366305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.245665] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.279367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1572.767320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1577.888113] LustreError: 55057:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 1604.512350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.287476] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1612.007419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1616.478876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.918504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.967301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1619.889892] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1624.035437] LustreError: 55981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.049183] LustreError: 55981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1626.641042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1626.908723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1630.387317] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1635.427795] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1638.266872] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1642.465777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1642.468197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.471650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.485845] Lustre: Skipped 1 previous similar message [ 1647.590427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1647.600791] Lustre: Skipped 2 previous similar messages [ 1652.711454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.711454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.241614] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1654.273133] LustreError: 57335:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1654.361560] Lustre: server umount lustre-OST0000 complete [ 1678.307450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.311540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.319258] Lustre: Skipped 1 previous similar message [ 1678.319643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1678.325107] Lustre: Skipped 1 previous similar message [ 1680.296567] LustreError: 57714:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1680.298796] LustreError: 57714:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1680.385973] Lustre: server umount lustre-MDT0000 complete [ 1683.428277] LustreError: 55986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1683.445586] LustreError: 55986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1683.660585] LustreError: 55066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334881 with bad export cookie 14698328829717662656 [ 1683.670405] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.675823] LustreError: 55066:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1689.899164] LustreError: 57916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1689.908959] LustreError: 57916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1690.144974] Lustre: server umount lustre-MDT0001 complete [ 1697.799652] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1699.777649] Key type lgssc unregistered [ 1700.004607] LNet: 58448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.016046] LNetError: 58448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.031701] LNet: Removed LNI 192.168.202.133@tcp [ 1701.567828] Key type .llcrypt unregistered [ 1701.569499] Key type ._llcrypt unregistered [ 1718.543706] Key type ._llcrypt registered [ 1718.545447] Key type .llcrypt registered [ 1718.637562] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1719.318796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.360402] alg: No test for adler32 (adler32-zlib) [ 1720.327823] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1720.497557] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1722.151294] Key type lgssc registered [ 1723.072723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1729.155510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1729.171788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.509379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.101554] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1739.572994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.748189] LustreError: 59820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1739.762596] LustreError: 59820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1739.985906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.553310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1745.944586] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.275055] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.918743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1753.160825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1755.241844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1757.492675] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1761.573631] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1781.222255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.225091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.237612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.241475] Lustre: Skipped 1 previous similar message [ 1784.140305] LustreError: 62090:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1784.293046] Lustre: server umount lustre-OST0000 complete [ 1791.455944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1791.460152] LustreError: Skipped 1 previous similar message [ 1791.460216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.465442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.469074] Lustre: Skipped 2 previous similar messages [ 1791.481335] Lustre: Skipped 1 previous similar message [ 1796.577087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.672584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.677543] Lustre: Skipped 2 previous similar messages [ 1802.207134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1802.282848] LustreError: 62293:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1802.306759] LustreError: 62293:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1802.386846] Lustre: server umount lustre-MDT0000 complete [ 1804.971156] LustreError: 61989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335002 with bad export cookie 17522525201722503040 [ 1804.971959] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.979543] LustreError: 61989:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1805.102217] LustreError: 62496:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1805.107443] LustreError: 62496:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1805.217162] Lustre: server umount lustre-MDT0001 complete [ 1812.568742] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1814.783699] Key type lgssc unregistered [ 1815.043727] LNet: 63028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.054581] LNetError: 63028:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.064929] LNet: Removed LNI 192.168.202.133@tcp [ 1815.712261] Key type .llcrypt unregistered [ 1815.714083] Key type ._llcrypt unregistered [ 1842.920020] Key type ._llcrypt registered [ 1842.921975] Key type .llcrypt registered [ 1843.028953] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1844.043356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.056321] alg: No test for adler32 (adler32-zlib) [ 1845.068529] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1845.247564] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1846.927266] Key type lgssc registered [ 1848.016835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1855.226648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1855.246660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.735335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.658662] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1867.236657] LustreError: 64714:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.254122] LustreError: 64714:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1869.026252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1869.594573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1873.251399] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1878.035084] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.991378] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1886.338404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1886.670818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.698745] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1895.068745] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1902.004641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1911.013602] Lustre: Failing over lustre-MDT0000 [ 1911.182645] LustreError: 66983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1911.342848] Lustre: server umount lustre-MDT0000 complete [ 1912.041425] LustreError: 64709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1912.289565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.308774] Lustre: Skipped 1 previous similar message [ 1917.120099] LustreError: 64710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.135378] LustreError: 64710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1919.105375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.200150] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.333670] Lustre: *** cfs_fail_loc=135, val=0*** [ 1919.398746] LustreError: 67494:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1919.401869] LustreError: 67494:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 1919.406378] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1919.406378] [ 1919.419464] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1919.450073] LustreError: 67482:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1919.472750] LustreError: 67482:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1919.487703] LustreError: 67482:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1919.519592] Lustre: server umount lustre-MDT0000 complete [ 1919.522916] LustreError: 67482:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1922.244547] LustreError: 66869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1922.265736] LustreError: 66869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1925.295692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.387761] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.552329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.601581] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1927.366438] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1929.087113] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1930.730195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1930.753180] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1930.773307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1935.847188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1935.851881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.864217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1935.866607] Lustre: Skipped 2 previous similar messages [ 1935.877844] Lustre: Skipped 1 previous similar message [ 1937.533720] LustreError: 68303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1937.547287] LustreError: 68303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1937.649936] Lustre: server umount lustre-OST0000 complete [ 1940.963172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.964156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1940.967677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.973734] Lustre: Skipped 1 previous similar message [ 1946.080868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.097622] Lustre: Skipped 2 previous similar messages [ 1946.851390] LustreError: 68504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1946.853896] LustreError: 68504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1946.933078] Lustre: server umount lustre-MDT0000 complete [ 1950.100202] LustreError: 64694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335147 with bad export cookie 15270128593439593124 [ 1950.111292] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.118381] LustreError: 64694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1950.235384] LustreError: 68705:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1950.244156] LustreError: 68705:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1950.482770] Lustre: server umount lustre-MDT0001 complete [ 1959.310843] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1961.147562] Key type lgssc unregistered [ 1961.391621] LNet: 69237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.399723] LNetError: 69237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.411798] LNet: Removed LNI 192.168.202.133@tcp [ 1961.953554] Key type .llcrypt unregistered [ 1961.955069] Key type ._llcrypt unregistered [ 1977.049626] Key type ._llcrypt registered [ 1977.054170] Key type .llcrypt registered [ 1977.126280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1977.672106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1977.688190] alg: No test for adler32 (adler32-zlib) [ 1978.574610] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 1978.694647] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1980.311185] Key type lgssc registered [ 1980.881601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1986.060239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1986.075192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.393102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.708931] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1995.886551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1996.162150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.989793] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2003.723601] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2006.732494] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2012.636916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2012.878633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.007549] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2023.101230] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.362322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2038.098697] Lustre: Failing over lustre-MDT0000 [ 2038.224861] LustreError: 72876:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2038.265866] Lustre: server umount lustre-MDT0000 complete [ 2040.017779] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.032705] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2042.336467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2042.344295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.356722] LustreError: 70609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.850264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.861656] Lustre: Skipped 1 previous similar message [ 2044.691399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.798499] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.892116] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2044.989204] LustreError: 73388:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2044.996900] LustreError: 73388:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -9 [ 2045.015522] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2045.015522] [ 2045.024802] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2045.038677] LustreError: 73376:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2045.051493] LustreError: 73376:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2045.061167] LustreError: 73376:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2045.091179] Lustre: server umount lustre-MDT0000 complete [ 2045.097268] LustreError: 73376:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2045.116390] LustreError: 71825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.127581] LustreError: 71825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2047.974332] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.989172] LustreError: 70604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2050.039415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2050.184300] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.237770] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (not set up) [ 2050.440380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.507876] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2053.241622] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2055.375643] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2055.654884] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2055.678860] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2055.703633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2060.769117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2060.774541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.776769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2060.802339] Lustre: Skipped 1 previous similar message [ 2064.767084] LustreError: 74200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2064.777495] LustreError: 74200:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2064.894170] Lustre: server umount lustre-OST0000 complete [ 2071.009223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.011664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.011924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.011928] Lustre: Skipped 1 previous similar message [ 2071.018348] Lustre: Skipped 1 previous similar message [ 2074.210641] LustreError: 74401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2074.214871] LustreError: 74401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2074.274579] Lustre: server umount lustre-MDT0000 complete [ 2076.132619] LustreError: 73203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.140701] LustreError: 73203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2077.142340] LustreError: 72877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335274 with bad export cookie 11800511066825010594 [ 2077.146686] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2077.157027] LustreError: 72877:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2077.309582] LustreError: 74603:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2077.335446] LustreError: 74603:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2077.574396] Lustre: server umount lustre-MDT0001 complete [ 2086.864130] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2088.897975] Key type lgssc unregistered [ 2089.107648] LNet: 75134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.118877] LNetError: 75134:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.136124] LNet: Removed LNI 192.168.202.133@tcp [ 2089.771198] Key type .llcrypt unregistered [ 2089.772852] Key type ._llcrypt unregistered [ 2098.477843] Key type ._llcrypt registered [ 2098.479457] Key type .llcrypt registered [ 2098.542357] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2098.731301] Key type .llcrypt unregistered [ 2098.732614] Key type ._llcrypt unregistered [ 2116.445038] Key type ._llcrypt registered [ 2116.446456] Key type .llcrypt registered [ 2116.551894] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2117.168556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2117.212718] alg: No test for adler32 (adler32-zlib) [ 2118.161981] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2118.315892] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2119.943184] Key type lgssc registered [ 2120.595769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2125.587380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2125.607248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.926668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.773456] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2136.030701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2136.354434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.415414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2143.739267] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2146.629952] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2152.000960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2152.282107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.176059] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2161.753282] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2171.757856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2182.114225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.115103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.117541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.148244] Lustre: Skipped 1 previous similar message [ 2183.858090] LustreError: 79245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2183.942248] Lustre: server umount lustre-OST0000 complete [ 2187.235700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.256037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2187.259501] Lustre: Skipped 2 previous similar messages [ 2192.354580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.795737] LustreError: 79447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2192.806372] LustreError: 79447:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2192.870530] Lustre: server umount lustre-MDT0000 complete [ 2195.805195] LustreError: 76956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335393 with bad export cookie 4052546073765296697 [ 2195.807960] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.823648] LustreError: 76956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.910245] LustreError: 79648:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2195.913539] LustreError: 79648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2196.038430] Lustre: server umount lustre-MDT0001 complete [ 2204.253240] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2206.455498] Key type lgssc unregistered [ 2206.712088] LNet: 80180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.715633] LNetError: 80180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.782075] LNet: Removed LNI 192.168.202.133@tcp [ 2208.353421] Key type .llcrypt unregistered [ 2208.355950] Key type ._llcrypt unregistered [ 2224.140000] Key type ._llcrypt registered [ 2224.141188] Key type .llcrypt registered [ 2224.213106] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2224.893325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.910645] alg: No test for adler32 (adler32-zlib) [ 2225.953316] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2226.094179] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2227.737546] Key type lgssc registered [ 2228.490680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2234.100459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2234.117833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.461415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.876922] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2245.606221] LustreError: 81553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.617540] LustreError: 81553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2246.258135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2246.567765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2250.339900] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2255.241071] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2258.751171] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2264.210694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2264.548696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.813604] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2274.987574] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2279.919790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2286.049099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2286.053828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.065843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.154510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.162889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.176991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.580378] LustreError: 83798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2291.699082] Lustre: server umount lustre-OST0000 complete [ 2295.265158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.269665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.282115] Lustre: Skipped 1 previous similar message [ 2295.302331] Lustre: Skipped 2 previous similar messages [ 2300.386894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.391237] Lustre: Skipped 1 previous similar message [ 2301.279777] LustreError: 84000:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2301.284844] LustreError: 84000:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2301.380281] Lustre: server umount lustre-MDT0000 complete [ 2304.527890] LustreError: 81532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335501 with bad export cookie 14580868906475655738 [ 2304.536978] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.550265] LustreError: 81532:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2304.697482] LustreError: 84202:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2304.713192] LustreError: 84202:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2304.885573] Lustre: server umount lustre-MDT0001 complete [ 2313.475223] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2315.682726] Key type lgssc unregistered [ 2315.954880] LNet: 84733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.966779] LNetError: 84733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.985274] LNet: Removed LNI 192.168.202.133@tcp [ 2316.684065] Key type .llcrypt unregistered [ 2316.685641] Key type ._llcrypt unregistered [ 2333.808392] Key type ._llcrypt registered [ 2333.810267] Key type .llcrypt registered [ 2333.885647] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2334.419638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.476345] alg: No test for adler32 (adler32-zlib) [ 2335.468719] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2335.668681] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2337.328402] Key type lgssc registered [ 2338.356736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2345.094988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2345.137211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2346.798772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.326993] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2352.096816] LustreError: 86121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.111641] LustreError: 86121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2357.223018] LustreError: 86121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2357.976982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2358.292823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2362.452379] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2367.802992] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2370.992059] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2376.565321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2376.947838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.179308] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2387.289050] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2392.303195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2402.791760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.792587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.802590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2402.811168] Lustre: Skipped 1 previous similar message [ 2405.833345] LustreError: 88388:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2405.953147] Lustre: server umount lustre-OST0000 complete [ 2409.440166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2409.443985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.452768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.455240] Lustre: Skipped 1 previous similar message [ 2413.027371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.029546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.148709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.151452] Lustre: Skipped 1 previous similar message [ 2423.263105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2423.265360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.280270] Lustre: Skipped 1 previous similar message [ 2423.354285] LustreError: 88589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2423.357555] LustreError: 88589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2423.452577] Lustre: server umount lustre-MDT0000 complete [ 2426.265759] LustreError: 88590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335623 with bad export cookie 17663916508843300259 [ 2426.274424] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.279524] LustreError: 88590:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2426.417939] LustreError: 88793:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2426.420989] LustreError: 88793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2426.550968] Lustre: server umount lustre-MDT0001 complete [ 2433.769451] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2435.588248] Key type lgssc unregistered [ 2435.772702] LNet: 89325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.778309] LNetError: 89325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.794808] LNet: Removed LNI 192.168.202.133@tcp [ 2436.396824] Key type .llcrypt unregistered [ 2436.402121] Key type ._llcrypt unregistered [ 2454.603228] Key type ._llcrypt registered [ 2454.604889] Key type .llcrypt registered [ 2454.704864] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2455.684095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.742056] alg: No test for adler32 (adler32-zlib) [ 2456.828315] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2457.076803] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2458.775177] Key type lgssc registered [ 2459.779842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2465.791337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2465.828340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2481.311284] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2486.431806] LustreError: 90669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 2513.119216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.812895] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2522.854848] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2527.456134] LustreError: 91526:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2527.537779] Lustre: server umount lustre-OST0000 complete [ 2533.447053] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:28:49 (1763335729) [ 2538.547361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.820502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2542.212410] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2544.101197] LustreError: 92197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.121223] LustreError: 92197:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2549.222467] LustreError: 92197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2550.179858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2550.530495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.994419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2559.054969] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.174692] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2568.966572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2569.419350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.115176] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2580.844501] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.731953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2596.320093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2596.330521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.354060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2599.391904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2599.402178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2599.403688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.417840] Lustre: Skipped 1 previous similar message [ 2602.104121] LustreError: 94604:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2602.107176] LustreError: 94604:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2602.241871] Lustre: server umount lustre-OST0000 complete [ 2607.075641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.080296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.087622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.633416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.633504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.617468] LustreError: 94806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2611.623926] LustreError: 94806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2611.692338] Lustre: server umount lustre-MDT0000 complete [ 2614.519814] LustreError: 92179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335811 with bad export cookie 5126980200336279370 [ 2614.525908] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2614.531721] LustreError: 92179:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2614.652829] LustreError: 95009:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2614.655252] LustreError: 95009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2614.775395] Lustre: server umount lustre-MDT0001 complete [ 2623.597625] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2625.872440] Key type lgssc unregistered [ 2626.153269] LNet: 95540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2626.161261] LNetError: 95540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2626.179785] LNet: Removed LNI 192.168.202.133@tcp [ 2626.772917] Key type .llcrypt unregistered [ 2626.774747] Key type ._llcrypt unregistered [ 2644.346849] Key type ._llcrypt registered [ 2644.348300] Key type .llcrypt registered [ 2644.446286] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2645.152643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.166160] alg: No test for adler32 (adler32-zlib) [ 2646.171283] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2646.345958] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2647.967264] Key type lgssc registered [ 2648.752655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2654.559054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2654.577580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.911416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2659.000330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2666.413632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2666.467486] LustreError: 96912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.486462] LustreError: 96912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2666.674695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2669.827514] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2674.032676] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2676.698338] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2682.167933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2682.451315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2687.130315] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2690.654768] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2700.904896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2708.799643] LustreError: 99181:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2708.881098] Lustre: server umount lustre-OST0000 complete [ 2711.521488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.527625] Lustre: Skipped 1 previous similar message [ 2711.532812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.640174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.644021] Lustre: Skipped 2 previous similar messages [ 2720.749566] LustreError: 99382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2720.752284] LustreError: 99382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2720.809960] Lustre: server umount lustre-MDT0000 complete [ 2721.762951] LustreError: 96911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.390863] LustreError: 96891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335920 with bad export cookie 9589702215601350563 [ 2723.401457] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.410153] LustreError: 96891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2733.035808] LustreError: 99586:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2733.040063] LustreError: 99586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2733.185705] Lustre: server umount lustre-MDT0001 complete [ 2740.025365] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2742.095551] Key type lgssc unregistered [ 2742.347985] LNet: 100118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.357791] LNetError: 100118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2742.371830] LNet: Removed LNI 192.168.202.133@tcp [ 2742.950233] Key type .llcrypt unregistered [ 2742.951926] Key type ._llcrypt unregistered [ 2756.499163] Key type ._llcrypt registered [ 2756.503976] Key type .llcrypt registered [ 2756.614731] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2757.446747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.510058] alg: No test for adler32 (adler32-zlib) [ 2758.550403] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 2758.741928] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2760.383179] Key type lgssc registered [ 2761.500490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.156150] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.178960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.687322] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2787.807743] LustreError: 101319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 2814.435668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.439910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2822.080194] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2827.053702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2827.249733] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2827.260874] LustreError: 102230:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2827.266541] LustreError: 102230:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2827.271184] LustreError: 102230:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2827.360973] Lustre: server umount lustre-MDT0000 complete [ 2827.364238] LustreError: 102230:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2843.616007] LustreError: 102661:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2843.728714] Lustre: server umount lustre-OST0000 complete [ 2850.399435] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2857.568878] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2864.764554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2870.396637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2875.728402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2881.744493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2887.806208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2887.872516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2888.123545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2888.188881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2888.280298] Lustre: lustre-MDT0000: new disk, initializing [ 2888.362830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.377135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2891.478380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2900.586977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2900.649951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2900.735570] Lustre: 105000:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2900.796131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2900.799357] Lustre: Skipped 1 previous similar message [ 2900.874278] Lustre: lustre-MDT0001: new disk, initializing [ 2900.999250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2901.050559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2901.064350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2903.947426] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2909.960170] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2913.322364] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2918.716948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2918.765507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2918.934931] Lustre: lustre-OST0000: new disk, initializing [ 2918.940548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2919.030204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.130941] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2926.126787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2926.135406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2926.206468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2928.940916] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2932.133176] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2932.331419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2935.236161] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2935.371642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2941.414169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.416608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2941.421105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2941.430370] Lustre: Skipped 1 previous similar message [ 2943.035608] LustreError: 107936:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2943.042054] LustreError: 107936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2943.175537] Lustre: server umount lustre-OST0000 complete [ 2946.534041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.538320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.540435] Lustre: Skipped 1 previous similar message [ 2946.544775] Lustre: Skipped 1 previous similar message [ 2951.650381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.659982] Lustre: Skipped 2 previous similar messages [ 2952.555868] LustreError: 108137:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2952.558024] LustreError: 108137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2952.634866] Lustre: server umount lustre-MDT0000 complete [ 2956.265081] LustreError: 104992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336153 with bad export cookie 7168023735152031215 [ 2956.268485] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2956.281063] LustreError: 104992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2956.446345] LustreError: 108338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2956.459863] LustreError: 108338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2956.697868] Lustre: server umount lustre-MDT0001 complete [ 2964.327535] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:35:59 (1763336159) [ 2967.161770] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2980.258665] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2988.762257] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2996.792116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3002.495837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.136600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.807098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.515865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3021.590680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3021.784935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3021.819982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3021.933396] Lustre: lustre-MDT0000: new disk, initializing [ 3022.062978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.091298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3026.014930] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3035.737416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3035.792856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3035.910416] Lustre: 111474:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3035.977152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3035.981270] Lustre: Skipped 1 previous similar message [ 3036.037269] Lustre: lustre-MDT0001: new disk, initializing [ 3036.120846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.163736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3036.178028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3040.286030] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3048.732406] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3052.232818] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3058.526496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3058.593626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3058.862597] Lustre: lustre-OST0000: new disk, initializing [ 3058.868125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3058.980759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3064.006256] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3068.495142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3068.508139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3068.653911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3070.991398] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3075.121958] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3075.376579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3079.680918] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3079.922406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3084.261863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.272575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.298638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.381744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.384289] Lustre: Skipped 1 previous similar message [ 3096.031137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3096.062247] LustreError: 114413:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3096.073730] LustreError: 114413:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3096.170591] Lustre: server umount lustre-OST0000 complete [ 3099.617794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.620128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3099.629390] Lustre: Skipped 2 previous similar messages [ 3099.632264] Lustre: Skipped 3 previous similar messages [ 3105.193936] LustreError: 114615:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3105.203980] LustreError: 114615:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3105.328250] Lustre: server umount lustre-MDT0000 complete [ 3108.462090] LustreError: 111466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336305 with bad export cookie 7168023735152032097 [ 3108.468836] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.486570] LustreError: 111466:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3108.756373] Lustre: server umount lustre-MDT0001 complete [ 3115.950099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.435807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.886121] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3121.639035] LustreError: 115296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.653090] LustreError: 115296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3125.737371] LustreError: 115296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.485640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.221253] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3135.321610] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3138.253921] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3144.369485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3150.064509] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3154.544429] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3170.640419] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3176.417363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3176.419762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3176.421583] LustreError: Skipped 1 previous similar message [ 3176.432542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3176.438694] Lustre: Skipped 4 previous similar messages [ 3180.041413] LustreError: 117660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3180.053245] LustreError: 117660:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3180.168888] Lustre: server umount lustre-OST0000 complete [ 3186.660752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3186.679026] Lustre: Skipped 1 previous similar message [ 3189.697771] Lustre: server umount lustre-MDT0000 complete [ 3191.777360] LustreError: 116358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.795131] LustreError: 116358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3192.718178] LustreError: 115276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336390 with bad export cookie 7168023735152032979 [ 3192.720736] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3192.729080] LustreError: 115276:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3193.029679] Lustre: server umount lustre-MDT0001 complete [ 3201.114556] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3203.518645] Key type lgssc unregistered [ 3203.778053] LNet: 118597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3203.785775] LNetError: 118597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3203.802145] LNet: Removed LNI 192.168.202.133@tcp [ 3204.618794] Key type .llcrypt unregistered [ 3204.620826] Key type ._llcrypt unregistered [ 3220.765105] Key type ._llcrypt registered [ 3220.766929] Key type .llcrypt registered [ 3220.865156] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 3232.294433] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3233.162524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3233.182375] alg: No test for adler32 (adler32-zlib) [ 3234.230410] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 3234.398277] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3236.039189] Key type lgssc registered [ 3237.028174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.556333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3250.529573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3256.277882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3262.713238] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3270.076065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3270.133658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3270.153770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.367236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3271.392346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3271.443309] Lustre: lustre-MDT0000: new disk, initializing [ 3271.542453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3271.552054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3274.628282] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3283.747344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3283.801674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3283.856921] Lustre: 121701:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3283.919565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3283.922373] Lustre: Skipped 1 previous similar message [ 3283.988217] Lustre: lustre-MDT0001: new disk, initializing [ 3284.107798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3284.163770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3284.169454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3287.647945] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3294.110961] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3296.689926] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3302.450994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3302.514822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3302.753583] Lustre: lustre-OST0000: new disk, initializing [ 3302.761941] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3302.849963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.805438] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3312.192247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3312.201369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3312.296472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3314.431185] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3318.379677] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3318.556700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3321.869402] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3322.006527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3327.969623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3327.970695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.978462] LustreError: Skipped 1 previous similar message [ 3327.981064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.988097] Lustre: Skipped 1 previous similar message [ 3329.463207] LustreError: 124641:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3329.588450] Lustre: server umount lustre-OST0000 complete [ 3333.090271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.097399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.104409] Lustre: Skipped 1 previous similar message [ 3333.108496] Lustre: Skipped 1 previous similar message [ 3338.212048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.214470] Lustre: Skipped 2 previous similar messages [ 3338.985376] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3338.988918] LustreError: 124845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3339.057455] Lustre: server umount lustre-MDT0000 complete [ 3342.498978] LustreError: 121695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336539 with bad export cookie 9031636134273535771 [ 3342.506593] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.519020] LustreError: 121695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3342.674065] LustreError: 125047:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3342.676318] LustreError: 125047:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3342.842370] Lustre: server umount lustre-MDT0001 complete [ 3350.337438] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:42:25 (1763336545) [ 3356.071892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.515917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.502744] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3361.763940] LustreError: 125717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.778817] LustreError: 125717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3365.862923] LustreError: 125717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3367.040099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3367.357499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3370.382738] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3375.150445] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3378.798898] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3382.752624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3382.755850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.767411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.827852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.981135] LustreError: 127061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3386.983289] LustreError: 127061:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3387.086737] Lustre: server umount lustre-MDT0000 complete [ 3390.306229] LustreError: 125698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336587 with bad export cookie 9031636134273536653 [ 3390.319230] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3390.688119] Lustre: server umount lustre-MDT0001 complete [ 3398.387272] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:43:13 (1763336593) [ 3404.193313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3419.808069] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3424.927468] LustreError: 127908:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3450.655307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.956337] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3460.444375] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3462.358847] LustreError: 128572:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3462.366953] LustreError: 128572:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3462.467109] Lustre: server umount lustre-OST0000 complete [ 3469.453803] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:44:25 (1763336665) [ 3474.821843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.200532] LustreError: 129244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.247412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.032576] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3480.548267] LustreError: 129244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.643343] LustreError: 129244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.755306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3487.945197] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3492.015717] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3494.864341] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3500.603700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.202990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3512.037392] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3530.720623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3530.729187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3530.731078] Lustre: Skipped 2 previous similar messages [ 3530.746610] Lustre: Skipped 4 previous similar messages [ 3534.005052] LustreError: 131508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3534.012504] LustreError: 131508:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3534.077662] Lustre: server umount lustre-MDT0000 complete [ 3535.842662] LustreError: 129240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.874737] LustreError: 129240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3537.148817] LustreError: 129224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336734 with bad export cookie 9031636134273537318 [ 3537.153405] LustreError: 129224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3537.158923] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.359072] Lustre: server umount lustre-MDT0001 complete [ 3550.669383] LustreError: 131910:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3550.677825] LustreError: 131910:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3550.869954] Lustre: server umount lustre-OST0000 complete [ 3559.694607] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:45:54 (1763336754) [ 3566.432178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.782787] LustreError: 132582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.880929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.885266] Lustre: Skipped 2 previous similar messages [ 3570.386497] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3578.606254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3582.222225] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3586.306561] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3589.435120] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3594.945355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3599.295180] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3603.152112] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3605.912427] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3611.756107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3612.338579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3614.767048] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3614.912802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3617.248717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3617.259500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3617.261239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3617.270944] Lustre: Skipped 1 previous similar message [ 3617.282024] Lustre: Skipped 1 previous similar message [ 3618.305290] LustreError: 135366:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3618.312262] LustreError: 135366:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3618.415154] Lustre: server umount lustre-OST0000 complete [ 3627.954379] Lustre: server umount lustre-MDT0000 complete [ 3630.930662] LustreError: 132562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336828 with bad export cookie 9031636134273538389 [ 3630.933139] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.950780] LustreError: 132562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3631.228668] Lustre: server umount lustre-MDT0001 complete [ 3638.522667] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:47:13 (1763336833) [ 3643.749795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3659.297060] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3664.415450] LustreError: 136413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3690.143603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.152944] Lustre: Skipped 2 previous similar messages [ 3694.183316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3698.689241] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3705.289298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3705.932921] LustreError: 136440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.943699] LustreError: 136440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3706.083203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3711.171793] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3720.270935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3724.092074] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3728.409573] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3731.727115] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3735.121245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3735.287893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3738.649316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3738.879470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3741.157357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3741.160889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.166659] Lustre: Skipped 2 previous similar messages [ 3741.169458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.176036] Lustre: Skipped 4 previous similar messages [ 3746.647310] LustreError: 139190:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3746.655979] LustreError: 139190:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3746.722665] Lustre: server umount lustre-MDT0000 complete [ 3749.457099] LustreError: 136421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336946 with bad export cookie 9031636134273539411 [ 3749.458304] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3749.465966] LustreError: 136421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3749.726557] Lustre: server umount lustre-MDT0001 complete [ 3763.187343] Lustre: server umount lustre-OST0000 complete [ 3771.706623] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:49:27 (1763336967) [ 3777.789686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.375558] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3798.495473] LustreError: 140241:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 3824.223597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.230646] Lustre: Skipped 2 previous similar messages [ 3828.032685] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3831.726107] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3836.122429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3836.433680] LustreError: 140268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.441843] LustreError: 140268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3836.539798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3839.161326] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3845.577470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3849.465610] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3852.983185] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3855.567774] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3860.884924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3860.940860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3861.065096] Lustre: lustre-OST0001: new disk, initializing [ 3861.068064] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3865.936078] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3868.216703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3868.229895] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3868.302643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3871.866155] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3874.740932] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3874.883467] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3878.111850] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3878.327968] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3881.957097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.966724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.978580] Lustre: Skipped 2 previous similar messages [ 3881.984601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3881.991702] Lustre: Skipped 3 previous similar messages [ 3894.248030] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3894.308522] LustreError: 143968:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3894.311961] LustreError: 143968:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3894.359541] Lustre: server umount lustre-OST0000 complete [ 3903.562157] Lustre: server umount lustre-OST0001 complete [ 3907.553308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3907.566412] LustreError: Skipped 1 previous similar message [ 3912.855520] Lustre: server umount lustre-MDT0000 complete [ 3916.132773] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337113 with bad export cookie 9031636134273540356 [ 3916.141081] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3916.150907] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3916.433791] Lustre: server umount lustre-MDT0001 complete [ 3920.735460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3923.869146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3926.883699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3930.085565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3935.541707] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:52:10 (1763337130) [ 3936.914200] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3938.086735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3939.537688] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:52:15 (1763337135) [ 3944.489692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.708956] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3944.732539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3947.873873] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3953.662798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3953.757884] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3953.777800] Lustre: 146216:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3957.125174] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3961.625808] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3964.344315] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3968.554565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3968.671115] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3972.286549] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3975.563695] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3977.787823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3978.247971] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3978.362168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3981.195875] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3981.359186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3982.821946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3982.822668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.831172] LustreError: Skipped 1 previous similar message [ 3982.840162] Lustre: Skipped 5 previous similar messages [ 3982.841843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.856223] Lustre: Skipped 13 previous similar messages [ 3985.012762] Lustre: server umount lustre-OST0000 complete [ 4004.491781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4009.338644] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4014.528740] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4018.618289] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4023.143944] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.154163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.157397] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4023.174091] Lustre: Skipped 1 previous similar message [ 4024.118161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4027.518574] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4027.760743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4031.064067] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8b5647e4a000.ost_server_uuid 50 [ 4032.493603] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8b5647e4a000.ost_server_uuid in FULL state after 0 sec [ 4032.659842] LustreError: 146224:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.33@tcp) returned error from blocking AST (req@00000000b4c20c3a x1848992261701120 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000001020b40/0x7d56d13663f4ea57 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0x70d34004327ae83a expref: 5 pid: 146223 timeout: 4132 lvb_type: 0 [ 4032.711153] LustreError: lustre-MDT0000: A client on nid 192.168.202.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4032.729906] LustreError: 146212:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000001020b40/0x7d56d13663f4ea57 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.33@tcp remote: 0x70d34004327ae83a expref: 6 pid: 146223 timeout: 0 lvb_type: 0 [ 4038.630632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.642585] LustreError: Skipped 1 previous similar message [ 4042.461567] Lustre: server umount lustre-OST0000 complete [ 4051.935362] Lustre: server umount lustre-MDT0000 complete [ 4053.985694] LustreError: 146228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4054.010710] LustreError: 146228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4054.935126] LustreError: 146208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337252 with bad export cookie 9031636134273541532 [ 4054.940454] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.947100] LustreError: 146208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4069.343325] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4069.626115] Lustre: server umount lustre-MDT0001 complete [ 4078.742376] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4080.905597] Key type lgssc unregistered [ 4081.126269] LNet: 151738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4081.134432] LNetError: 151738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4081.153639] LNet: Removed LNI 192.168.202.133@tcp [ 4081.889498] Key type .llcrypt unregistered [ 4081.896911] Key type ._llcrypt unregistered [ 4100.840155] Key type ._llcrypt registered [ 4100.842379] Key type .llcrypt registered [ 4100.917018] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4101.650298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4101.667310] alg: No test for adler32 (adler32-zlib) [ 4102.648977] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4102.858581] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4104.479207] Key type lgssc registered [ 4105.556417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4111.794156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4111.826642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.250829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4115.709924] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4121.371474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4121.660976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4124.147910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4127.807613] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4130.194211] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4134.814956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4135.178974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.288614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4139.783470] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4143.994747] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4149.495685] Lustre: Failing over lustre-MDT0000 [ 4149.620559] LustreError: 155354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4149.676372] Lustre: server umount lustre-MDT0000 complete [ 4151.781362] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.786258] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.794971] Lustre: Skipped 1 previous similar message [ 4151.804595] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4156.901162] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.920801] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4161.430698] LustreError: 153091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.33@tcp arrived at 1763337358 with bad export cookie 1004795959729639414 [ 4161.453497] LustreError: 153091:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4162.017387] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4162.050333] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4167.139462] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.161741] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4168.167791] Lustre: 152420:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337349/real 1763337349] req@0000000082022d3c x1848993172706816/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763337365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4168.194255] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4168.649089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4173.283329] LustreError: 153111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.295346] LustreError: 153111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4178.402979] LustreError: 152418:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008d151f06 x1848993172712832/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4178.559175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.629610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4181.575465] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4183.469583] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4183.477155] Lustre: lustre-MDT0000: Denying connection for new client 1e0c281a-52c5-4a7c-986d-9e807f4f3fc8 (at 192.168.202.33@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4184.051171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4192.739440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4192.747979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4192.770225] Lustre: Skipped 1 previous similar message [ 4192.773705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.278481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.280752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.570143] LustreError: 156288:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4197.572676] LustreError: 156288:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4197.663276] Lustre: server umount lustre-OST0000 complete [ 4200.886115] LustreError: 156490:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4200.891506] Lustre: 155855:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4200.899627] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4200.912382] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4200.919141] LustreError: 155855:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000028066137 x1848993172726272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4200.928618] LustreError: 155855:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4200.933541] LustreError: 155855:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4200.941902] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 4201.069934] LustreError: 156490:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4201.086033] LustreError: 156490:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4201.170989] Lustre: server umount lustre-MDT0000 complete [ 4204.433424] LustreError: 155355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337401 with bad export cookie 1004795959729639519 [ 4204.437520] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.451959] LustreError: 155355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4204.461790] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.480807] LustreError: 153110:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4210.642853] LustreError: 156691:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4210.646565] LustreError: 156691:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4210.751510] Lustre: server umount lustre-MDT0001 complete [ 4218.753856] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4221.192682] Key type lgssc unregistered [ 4221.448372] LNet: 157224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4221.456712] LNetError: 157224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4221.469220] LNet: Removed LNI 192.168.202.133@tcp [ 4222.121826] Key type .llcrypt unregistered [ 4222.123764] Key type ._llcrypt unregistered [ 4241.123717] Key type ._llcrypt registered [ 4241.125204] Key type .llcrypt registered [ 4241.218946] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4241.868155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4241.916858] alg: No test for adler32 (adler32-zlib) [ 4242.968070] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4243.140303] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4244.799362] Key type lgssc registered [ 4245.744617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4253.121411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4253.136025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4254.669650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.799480] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4263.916167] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.939721] LustreError: 158597:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4267.097929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4267.344215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4270.797222] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4276.589226] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4280.436552] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4286.982912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4287.227473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.686825] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4297.809567] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4299.516695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4307.942425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4307.949862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4307.972860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4309.992529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4309.996884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4310.004691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.036118] Lustre: Skipped 1 previous similar message [ 4313.856112] LustreError: 160845:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4314.003678] Lustre: server umount lustre-OST0000 complete [ 4318.688817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.691932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.698286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4323.808838] LustreError: 161047:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4323.811134] LustreError: 161047:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4323.913478] Lustre: server umount lustre-MDT0000 complete [ 4325.349130] LustreError: 160755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.367339] LustreError: 160755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4327.097598] LustreError: 158577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337524 with bad export cookie 3116197507729735911 [ 4327.099667] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.110905] LustreError: 158577:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4327.240253] LustreError: 161249:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4327.242581] LustreError: 161249:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4327.477969] Lustre: server umount lustre-MDT0001 complete [ 4335.657103] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4337.951662] Key type lgssc unregistered [ 4338.208716] LNet: 161782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4338.218063] LNetError: 161782:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4338.233814] LNet: Removed LNI 192.168.202.133@tcp [ 4338.940708] Key type .llcrypt unregistered [ 4338.945407] Key type ._llcrypt unregistered [ 4368.085024] Key type ._llcrypt registered [ 4368.087977] Key type .llcrypt registered [ 4368.184404] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4368.927833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4368.949943] alg: No test for adler32 (adler32-zlib) [ 4369.950163] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4370.265625] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4371.935134] Key type lgssc registered [ 4372.941682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4379.718995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4379.737841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.099656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.573431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4391.409196] LustreError: 163467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.416505] LustreError: 163467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4393.253583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4393.726294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.714512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4403.070114] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4406.882221] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.795327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4414.276394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.930926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4425.743996] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4426.557397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4431.845320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4431.850956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.866097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.666199] LustreError: 165710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4433.773543] Lustre: server umount lustre-OST0000 complete [ 4436.961897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4436.968593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4436.968888] Lustre: Skipped 2 previous similar messages [ 4436.977488] Lustre: Skipped 2 previous similar messages [ 4442.084748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4442.095491] Lustre: Skipped 1 previous similar message [ 4447.203109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.295223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.395425] LustreError: 165913:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4451.409518] LustreError: 165913:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4451.584349] Lustre: server umount lustre-MDT0000 complete [ 4452.322875] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.349935] LustreError: 163466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4455.358229] LustreError: 163448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337652 with bad export cookie 11682173826187156211 [ 4455.360600] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.390294] LustreError: 163448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4455.544535] LustreError: 166117:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4455.547239] LustreError: 166117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4455.887684] Lustre: server umount lustre-MDT0001 complete [ 4467.253324] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4470.006898] Key type lgssc unregistered [ 4470.370099] LNet: 166659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4470.379642] LNetError: 166659:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4470.432591] LNet: Removed LNI 192.168.202.133@tcp [ 4471.209206] Key type .llcrypt unregistered [ 4471.210980] Key type ._llcrypt unregistered [ 4493.527518] Key type ._llcrypt registered [ 4493.529182] Key type .llcrypt registered [ 4493.730581] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4494.800047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4494.811295] alg: No test for adler32 (adler32-zlib) [ 4496.117813] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4496.379059] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4498.103820] Key type lgssc registered [ 4499.849901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4511.275289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4511.304097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.748701] Lustre: *** cfs_fail_loc=135, val=0*** [ 4512.893107] LustreError: 168165:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4512.906197] LustreError: 168165:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.133@tcp: cfg command failed: rc = -2 [ 4512.928741] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4512.928741] [ 4512.933596] LustreError: MGC192.168.202.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4512.946841] LustreError: 168125:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4512.951551] LustreError: 168125:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4512.958488] LustreError: 168125:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4512.967779] LustreError: 168125:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4513.070181] Lustre: server umount lustre-MDT0000 complete [ 4513.077988] LustreError: 168125:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4519.770384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.397952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.406627] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4529.642205] LustreError: 168537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.750376] LustreError: 168537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4533.309419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4533.679148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4537.446786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4543.045728] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4546.493545] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4549.089392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4549.098873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.108104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.114117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4550.118206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.439173] LustreError: 169881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4554.443467] LustreError: 169881:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4554.616311] Lustre: server umount lustre-MDT0000 complete [ 4555.238201] LustreError: 169883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.266822] LustreError: 169883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4558.305038] LustreError: 168538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.317020] LustreError: 168517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337755 with bad export cookie 769054302767828480 [ 4558.335743] LustreError: 168538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4558.343984] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4558.491172] LustreError: 170084:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4558.495248] LustreError: 170084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4558.688065] Lustre: server umount lustre-MDT0001 complete [ 4567.408173] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4569.795525] Key type lgssc unregistered [ 4570.040793] LNet: 170616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4570.055298] LNetError: 170616:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4570.069680] LNet: Removed LNI 192.168.202.133@tcp [ 4570.787588] Key type .llcrypt unregistered [ 4570.789325] Key type ._llcrypt unregistered [ 4589.106431] Key type ._llcrypt registered [ 4589.108360] Key type .llcrypt registered [ 4589.199493] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4589.443379] Key type .llcrypt unregistered [ 4589.449330] Key type ._llcrypt unregistered [ 4604.178101] Key type ._llcrypt registered [ 4604.179678] Key type .llcrypt registered [ 4604.267969] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4605.304252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4605.363067] alg: No test for adler32 (adler32-zlib) [ 4606.362878] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4606.535551] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4608.183180] Key type lgssc registered [ 4609.207669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4615.585960] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4615.620962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4631.200688] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4636.319231] LustreError: 172706:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 4662.943538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.412527] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4673.095022] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4679.689161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.328293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.377786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4684.293632] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4685.798190] LustreError: 172732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.830756] LustreError: 172732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4690.917085] LustreError: 172732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.703573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4693.244791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4697.210862] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4702.018338] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4705.892587] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4708.761543] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4719.072423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.077045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.087911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.089573] Lustre: Skipped 1 previous similar message [ 4723.712411] LustreError: 175334:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4723.873876] Lustre: server umount lustre-OST0000 complete [ 4729.312559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4729.313065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.327819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.346348] Lustre: Skipped 1 previous similar message [ 4729.374809] Lustre: Skipped 2 previous similar messages [ 4734.435584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4734.449429] Lustre: Skipped 1 previous similar message [ 4739.554595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4739.560797] Lustre: Skipped 1 previous similar message [ 4742.111140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4742.208475] LustreError: 175535:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4742.224962] LustreError: 175535:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4742.442609] Lustre: server umount lustre-MDT0000 complete [ 4744.673990] LustreError: 173637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4744.704543] LustreError: 173637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4746.270631] LustreError: 172714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337943 with bad export cookie 17635654710930102075 [ 4746.278747] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.281386] LustreError: 172714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4746.431145] LustreError: 175738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4746.442824] LustreError: 175738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4746.674260] Lustre: server umount lustre-MDT0001 complete [ 4757.531885] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4760.507817] Key type lgssc unregistered [ 4760.969473] LNet: 176270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4760.988118] LNetError: 176270:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4762.026614] LNet: Removed LNI 192.168.202.133@tcp [ 4762.982079] Key type .llcrypt unregistered [ 4762.983717] Key type ._llcrypt unregistered [ 4786.936417] Key type ._llcrypt registered [ 4786.938189] Key type .llcrypt registered [ 4787.028645] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4788.420387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4788.432941] alg: No test for adler32 (adler32-zlib) [ 4789.630242] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4790.006760] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4791.696372] Key type lgssc registered [ 4792.919337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4799.645621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4799.682319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4801.131929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.634779] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4810.720258] LustreError: 177666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.733921] LustreError: 177666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4812.161323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4812.587618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4816.123328] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4821.245645] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4824.603794] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.753891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4831.181174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.296598] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4842.490711] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4846.506331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4851.582416] Lustre: Failing over lustre-MDT0000 [ 4851.684790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.688202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.703310] Lustre: Skipped 2 previous similar messages [ 4851.710282] Lustre: Skipped 1 previous similar message [ 4851.741211] LustreError: 179967:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4851.864234] Lustre: server umount lustre-MDT0000 complete [ 4853.464068] LustreError: 179831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.805242] LustreError: 179831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4856.834341] LustreError: 179831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4861.922571] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.960934] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4867.049338] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4867.074634] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4869.960931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.044625] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4870.270320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4870.340693] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4873.849673] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4873.928095] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4875.754823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4875.775861] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4875.817971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4877.836882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4879.369315] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4893.664390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4893.673980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.685404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.688311] Lustre: Skipped 1 previous similar message [ 4895.210328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4895.216437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.230075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.748438] LustreError: 181135:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4898.750370] LustreError: 181135:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4898.892543] Lustre: server umount lustre-OST0000 complete [ 4906.464830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4906.466495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.488977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4906.492339] Lustre: Skipped 1 previous similar message [ 4906.513060] Lustre: Skipped 3 previous similar messages [ 4911.589337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4916.191234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4916.360846] LustreError: 181338:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4916.365165] LustreError: 181338:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4916.486383] Lustre: server umount lustre-MDT0000 complete [ 4916.724370] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.763971] LustreError: 177667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4920.197051] LustreError: 177646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338117 with bad export cookie 17530462188238132971 [ 4920.198957] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4920.209955] LustreError: 177646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4920.334218] LustreError: 181540:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4920.339838] LustreError: 181540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4920.481668] Lustre: server umount lustre-MDT0001 complete [ 4930.812515] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4933.344452] Key type lgssc unregistered [ 4933.612747] LNet: 182072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4933.624493] LNetError: 182072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4933.647304] LNet: Removed LNI 192.168.202.133@tcp [ 4934.383405] Key type .llcrypt unregistered [ 4934.386412] Key type ._llcrypt unregistered [ 4953.653247] Key type ._llcrypt registered [ 4953.659357] Key type .llcrypt registered [ 4953.729645] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4954.328077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4954.337738] alg: No test for adler32 (adler32-zlib) [ 4955.350529] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 4955.549751] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4957.183204] Key type lgssc registered [ 4958.054706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.368723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.388367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.790695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.446255] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4970.980618] LustreError: 183445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.999336] LustreError: 183445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4976.104862] LustreError: 183445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4977.886458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4978.162273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4981.643617] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4986.668119] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4990.175138] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4995.653236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4995.973505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.516635] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5005.618391] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5011.314974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5012.692115] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5023.760724] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5033.393372] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5044.193660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5044.210283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.229501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.266268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5047.268207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5047.269665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.081578] LustreError: 185879:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5049.208322] Lustre: server umount lustre-OST0000 complete [ 5054.947550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5054.951128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.958324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.961210] Lustre: Skipped 1 previous similar message [ 5057.505440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.505678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5062.629021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5062.631736] Lustre: Skipped 2 previous similar messages [ 5067.231128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5067.307438] LustreError: 186081:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5067.310081] LustreError: 186081:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5067.370171] Lustre: server umount lustre-MDT0000 complete [ 5067.744668] LustreError: 183445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5067.760295] LustreError: 183445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5070.121767] LustreError: 183423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338267 with bad export cookie 13940374455272095952 [ 5070.125234] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.132750] LustreError: 183423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5070.248779] LustreError: 186283:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5070.257157] LustreError: 186283:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5070.397685] Lustre: server umount lustre-MDT0001 complete [ 5077.933452] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5079.812366] Key type lgssc unregistered [ 5080.068320] LNet: 186815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5080.079389] LNetError: 186815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5080.092593] LNet: Removed LNI 192.168.202.133@tcp [ 5080.686933] Key type .llcrypt unregistered [ 5080.688490] Key type ._llcrypt unregistered [ 5099.357660] Key type ._llcrypt registered [ 5099.359532] Key type .llcrypt registered [ 5099.434334] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5100.164762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5100.182503] alg: No test for adler32 (adler32-zlib) [ 5101.201261] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5101.393775] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5103.039176] Key type lgssc registered [ 5103.931488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5110.465140] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5110.482252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.910879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.207467] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5122.528794] LustreError: 188187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.549394] LustreError: 188187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5122.948054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5123.385828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.145238] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5131.426066] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5134.303219] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5139.256296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5139.598240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5143.898475] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5147.733373] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5159.084684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5163.189782] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5175.092388] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5182.149707] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5186.922592] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5199.329065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.332984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.340662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5200.353203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5200.356171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.371671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5205.478059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.487924] Lustre: Skipped 2 previous similar messages [ 5210.079160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5210.108221] LustreError: 191899:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5210.158946] Lustre: server umount lustre-OST0000 complete [ 5215.712158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5215.715862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.725918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5215.734295] Lustre: Skipped 1 previous similar message [ 5220.834303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.838246] Lustre: Skipped 2 previous similar messages [ 5226.976510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5227.027450] LustreError: 192102:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5227.030437] LustreError: 192102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5227.070093] Lustre: server umount lustre-MDT0000 complete [ 5229.853601] LustreError: 188167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338427 with bad export cookie 5054261161864102558 [ 5229.859264] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5229.900841] LustreError: 188167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5230.019308] LustreError: 192304:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5230.022436] LustreError: 192304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5230.136684] Lustre: server umount lustre-MDT0001 complete [ 5238.959356] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5241.072815] Key type lgssc unregistered [ 5241.297857] LNet: 192835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5241.303746] LNetError: 192835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5241.324524] LNet: Removed LNI 192.168.202.133@tcp [ 5241.895855] Key type .llcrypt unregistered [ 5241.897404] Key type ._llcrypt unregistered [ 5260.522767] Key type ._llcrypt registered [ 5260.524313] Key type .llcrypt registered [ 5260.598994] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5261.363144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.371744] alg: No test for adler32 (adler32-zlib) [ 5262.369424] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5262.560765] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5264.207190] Key type lgssc registered [ 5265.110315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5271.676708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5271.715296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.322107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5276.929169] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5278.689082] LustreError: 194208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.715614] LustreError: 194208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5282.785845] LustreError: 194208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.895381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5284.144035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.410937] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5293.447045] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5297.373467] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5305.417349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5305.928982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5312.434293] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5317.518786] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5318.279568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5333.984135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.990987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.000664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.006182] Lustre: Skipped 1 previous similar message [ 5334.013456] Lustre: Skipped 1 previous similar message [ 5336.774151] LustreError: 196607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5336.944387] Lustre: server umount lustre-OST0000 complete [ 5340.640395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5340.652375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.658890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.224921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.226765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.611254] LustreError: 196811:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5346.621285] LustreError: 196811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5346.749406] Lustre: server umount lustre-MDT0000 complete [ 5349.344567] LustreError: 194232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.354796] LustreError: 194232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5350.154526] LustreError: 194189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338547 with bad export cookie 6462173563667024874 [ 5350.155690] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.163149] LustreError: 194189:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5350.270218] LustreError: 197012:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5350.279405] LustreError: 197012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5350.400206] Lustre: server umount lustre-MDT0001 complete [ 5358.419733] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5360.435944] Key type lgssc unregistered [ 5360.662347] LNet: 197544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5360.672413] LNetError: 197544:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5360.686448] LNet: Removed LNI 192.168.202.133@tcp [ 5361.233793] Key type .llcrypt unregistered [ 5361.236523] Key type ._llcrypt unregistered [ 5375.288722] Key type ._llcrypt registered [ 5375.290338] Key type .llcrypt registered [ 5375.391566] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5376.092769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5376.122287] alg: No test for adler32 (adler32-zlib) [ 5377.150540] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5377.303960] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5378.943194] Key type lgssc registered [ 5379.902134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5386.463106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5386.486556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.984490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.335620] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5398.258082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5398.534732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5401.596637] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5406.097452] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.500979] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5415.553110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5415.932752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.243760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5427.122772] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5434.354334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5449.698561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5449.702461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.708326] Lustre: Skipped 1 previous similar message [ 5449.712140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.722682] Lustre: Skipped 1 previous similar message [ 5454.816974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5454.825511] Lustre: Skipped 1 previous similar message [ 5455.363106] LustreError: 201198:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5455.528858] Lustre: server umount lustre-OST0000 complete [ 5459.939234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.941530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.965515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5464.872208] LustreError: 201400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5464.883271] LustreError: 201400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5465.000571] Lustre: server umount lustre-MDT0000 complete [ 5465.061717] LustreError: 198769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.070422] LustreError: 198769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5467.979869] LustreError: 198750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338665 with bad export cookie 6790131246264370924 [ 5467.987245] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5467.995047] LustreError: 198750:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5468.016697] LustreError: 198082:0:(import.c:716:ptlrpc_connect_import_locked()) already connecting [ 5468.149233] LustreError: 201602:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5468.152360] LustreError: 201602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5468.381672] Lustre: server umount lustre-MDT0001 complete [ 5478.762593] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5481.425072] Key type lgssc unregistered [ 5481.720546] LNet: 202136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5481.727389] LNetError: 202136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.742072] LNet: Removed LNI 192.168.202.133@tcp [ 5482.427885] Key type .llcrypt unregistered [ 5482.431259] Key type ._llcrypt unregistered [ 5499.964952] Key type ._llcrypt registered [ 5499.966584] Key type .llcrypt registered [ 5500.073168] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5501.235733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5501.259494] alg: No test for adler32 (adler32-zlib) [ 5502.272857] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5502.604936] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5504.311341] Key type lgssc registered [ 5505.577127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5513.647962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5513.668748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.033517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5518.254441] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5525.477683] LustreError: 203381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.486719] LustreError: 203381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5526.323335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5526.786462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.128587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5535.472437] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5538.867313] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5546.709346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5547.180783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.455406] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5558.317681] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5560.498900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5570.127735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5570.180672] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5570.344627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5574.865711] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5579.177839] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5580.363966] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5591.771366] Lustre: Permanently deactivating lustre-OST0001 [ 5591.794229] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5601.332142] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5608.675928] Lustre: Permanently reactivating lustre-OST0001 [ 5608.694370] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5608.704442] Lustre: Skipped 2 previous similar messages [ 5609.004598] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.013311] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5609.024805] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5609.037819] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5609.070250] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5621.883630] LustreError: 206879:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5621.954169] Lustre: server umount lustre-OST0001 complete [ 5622.248028] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.261765] Lustre: Skipped 1 previous similar message [ 5622.266804] LustreError: 205543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.283262] LustreError: 205543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5627.360489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.365610] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.366313] LustreError: 206498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5627.374436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5627.411219] Lustre: Skipped 2 previous similar messages [ 5631.164690] LustreError: 207081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5631.173026] LustreError: 207081:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5631.327325] Lustre: server umount lustre-OST0000 complete [ 5634.529517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5634.536219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.543117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.546288] Lustre: Skipped 1 previous similar message [ 5637.607611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.615766] Lustre: Skipped 1 previous similar message [ 5640.689729] LustreError: 207282:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5640.691611] LustreError: 207282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5640.765901] Lustre: server umount lustre-MDT0000 complete [ 5642.720726] LustreError: 205544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5642.739628] LustreError: 205544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5644.144696] LustreError: 205891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338841 with bad export cookie 8075528505969072882 [ 5644.147450] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5644.160556] LustreError: 205891:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5644.360754] LustreError: 207484:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5644.367618] LustreError: 207484:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5644.589046] Lustre: server umount lustre-MDT0001 complete [ 5654.082870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 5656.639474] Key type lgssc unregistered [ 5656.881313] LNet: 208016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.889522] LNetError: 208016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5657.961806] LNet: Removed LNI 192.168.202.133@tcp [ 5658.662430] Key type .llcrypt unregistered [ 5658.664329] Key type ._llcrypt unregistered [ 5685.468873] Key type ._llcrypt registered [ 5685.472754] Key type .llcrypt registered [ 5685.541217] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5699.879393] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5700.954531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5701.060043] alg: No test for adler32 (adler32-zlib) [ 5702.118077] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 5702.423165] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 5704.087270] Key type lgssc registered [ 5705.278504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5713.606967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5719.677228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.865980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5730.657179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5737.050582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5737.125951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5737.136107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5738.367493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5738.397910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5738.483290] Lustre: lustre-MDT0000: new disk, initializing [ 5738.592584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5738.613396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5742.106813] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5752.396528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5752.465826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5752.552495] Lustre: 211960:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5752.623152] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5752.625632] Lustre: Skipped 1 previous similar message [ 5752.699794] Lustre: lustre-MDT0001: new disk, initializing [ 5752.826079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5752.892296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5752.908845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5756.954937] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5764.212065] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5767.674812] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5774.133984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5774.203221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5774.450215] Lustre: lustre-OST0000: new disk, initializing [ 5774.462442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5774.555429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5779.929681] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5780.567186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5780.582910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5780.722195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5788.497944] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5792.885739] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5793.129609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5797.372190] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5797.555954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5801.441244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.442265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5801.442478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5801.446542] Lustre: Skipped 1 previous similar message [ 5805.568741] LustreError: 214899:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5805.708492] Lustre: server umount lustre-OST0000 complete [ 5811.686375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.697467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5811.701453] Lustre: Skipped 1 previous similar message [ 5816.281510] LustreError: 215100:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5816.284619] LustreError: 215100:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5816.346094] Lustre: server umount lustre-MDT0000 complete [ 5820.397348] LustreError: 211953:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339017 with bad export cookie 4537503225318406332 [ 5820.403988] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5820.608934] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5820.615593] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5820.754259] Lustre: server umount lustre-MDT0001 complete [ 5830.027905] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 19:23:45 (1763339025) [ 5836.585818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.989225] LustreError: 215974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.000022] LustreError: 215974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5837.063340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.998513] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5842.402161] LustreError: 215974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.412519] LustreError: 215974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5846.668790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5847.109589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.513649] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5855.122244] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5858.377892] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5865.049184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5865.373022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.152646] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5876.033380] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5888.792173] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5895.283654] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5902.856757] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5912.555805] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5922.177228] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5947.131504] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5947.135287] Lustre: Skipped 2 previous similar messages [ 5979.928457] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5979.936679] Lustre: Skipped 3 previous similar messages [ 6046.246022] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6046.251137] Lustre: Skipped 7 previous similar messages [ 6077.408369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6077.414813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6077.424024] LustreError: Skipped 1 previous similar message [ 6077.424665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6077.457366] Lustre: Skipped 2 previous similar messages [ 6078.528077] LustreError: 219463:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6078.545513] LustreError: 219463:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6078.661539] Lustre: server umount lustre-OST0000 complete [ 6082.532465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.532864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6082.533459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.533463] Lustre: Skipped 1 previous similar message [ 6082.544337] Lustre: Skipped 1 previous similar message [ 6087.648287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.650819] Lustre: Skipped 2 previous similar messages [ 6088.437241] LustreError: 219668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6088.441797] LustreError: 219668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6088.580211] Lustre: server umount lustre-MDT0000 complete [ 6091.982340] LustreError: 215956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339289 with bad export cookie 4537503225318407214 [ 6091.984257] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.992182] LustreError: 215956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6092.412704] Lustre: server umount lustre-MDT0001 complete [ 6103.539944] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6106.677466] Key type lgssc unregistered [ 6106.949242] LNet: 220403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6106.962135] LNetError: 220403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6106.978649] LNet: Removed LNI 192.168.202.133@tcp [ 6107.850769] Key type .llcrypt unregistered [ 6107.852580] Key type ._llcrypt unregistered [ 6130.456508] Key type ._llcrypt registered [ 6130.464300] Key type .llcrypt registered [ 6130.629582] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6131.938377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6132.051395] alg: No test for adler32 (adler32-zlib) [ 6133.202372] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 6133.479521] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6135.175183] Key type lgssc registered [ 6136.286361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.662662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.676763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6146.157767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.959771] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6155.763543] LustreError: 221797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.790174] LustreError: 221797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6158.224758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6158.589278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6162.595845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6168.569706] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6172.878785] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6179.981295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6180.429462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6186.656594] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6193.153301] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6211.555456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6211.556850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.558708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6211.578332] Lustre: Skipped 1 previous similar message [ 6211.719230] LustreError: 224204:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6211.857795] Lustre: server umount lustre-OST0000 complete [ 6216.673033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.684228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.686847] Lustre: Skipped 1 previous similar message [ 6221.793181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6221.798906] Lustre: Skipped 1 previous similar message [ 6222.082058] LustreError: 224405:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6222.094334] LustreError: 224405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6222.287452] Lustre: server umount lustre-MDT0000 complete [ 6225.932293] LustreError: 221780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339423 with bad export cookie 1560611088898329267 [ 6225.935205] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.950523] LustreError: 221780:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6226.075589] LustreError: 224606:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6226.094265] LustreError: 224606:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6226.317553] Lustre: server umount lustre-MDT0001 complete [ 6237.025439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6239.852854] Key type lgssc unregistered [ 6240.158742] LNet: 225139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6240.170680] LNetError: 225139:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6240.195109] LNet: Removed LNI 192.168.202.133@tcp [ 6240.969488] Key type .llcrypt unregistered [ 6240.970699] Key type ._llcrypt unregistered [ 6261.265583] Key type ._llcrypt registered [ 6261.266714] Key type .llcrypt registered [ 6261.367586] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6261.671611] Key type .llcrypt unregistered [ 6261.673321] Key type ._llcrypt unregistered [ 6287.927636] Key type ._llcrypt registered [ 6287.929409] Key type .llcrypt registered [ 6288.095741] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6303.443256] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6304.388317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6304.411254] alg: No test for adler32 (adler32-zlib) [ 6305.458334] Lustre: Lustre: Build Version: 2.16.61_52_gd3487c4 [ 6305.670996] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6307.343128] Key type lgssc registered [ 6308.383610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6316.565868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6322.213976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6328.286987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6335.192675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6336.914594] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6115 sec ========== 19:32:12 (1763339532) [ 6338.186546] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:32:13 (1763339533) === [ 6341.770299] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:32:16 (1763339536) === [ 6379.744622] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6382.400989] Key type lgssc unregistered [ 6382.627595] LNet: 231251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6382.634636] LNetError: 231251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6382.650621] LNet: Removed LNI 192.168.202.133@tcp [ 6383.262675] Key type .llcrypt unregistered [ 6383.265670] Key type ._llcrypt unregistered