[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491824940 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003271] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008356] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012068] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014102] *** VALIDATE selinux *** [ 0.022672] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027155] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] *** VALIDATE tmpfs *** [ 0.032471] *** VALIDATE proc *** [ 0.033229] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036214] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044234] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.046965] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054018] ... event mask: 000000070000000f [ 0.055327] rcu: Hierarchical SRCU implementation. [ 0.057465] smp: Bringing up secondary CPUs ... [ 0.058639] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.064024] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152473] node 0 deferred pages initialised in 81ms [ 0.157241] devtmpfs: initialized [ 0.158291] x86/mm: Memory block size: 128MB [ 0.161000] gcov: version magic: 0x41383552 [ 0.163249] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.167112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.169353] pinctrl core: initialized pinctrl subsystem [ 0.171190] [ 0.171721] ************************************************************* [ 0.174012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ** ** [ 0.178016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180011] ** ** [ 0.182009] ** This means that this kernel is built to expose internal ** [ 0.184012] ** IOMMU data structures, which may compromise security on ** [ 0.186011] ** your system. ** [ 0.188013] ** ** [ 0.190013] ** If you see this message and you are not debugging the ** [ 0.192013] ** kernel, report this immediately to your vendor! ** [ 0.194012] ** ** [ 0.195012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197015] ************************************************************* [ 0.199641] NET: Registered protocol family 16 [ 0.201428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210464] cpuidle: using governor menu [ 0.211756] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212698] PCI: Using configuration type 1 for base access [ 0.213164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220134] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.226161] cryptd: max_cpu_qlen set to 1000 [ 0.229291] ACPI: Added _OSI(Module Device) [ 0.230022] ACPI: Added _OSI(Processor Device) [ 0.231028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240605] ACPI: Interpreter enabled [ 0.241000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241000] ACPI: Using IOAPIC for interrupt routing [ 0.243193] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246546] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258092] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271426] acpiphp: Slot [2] registered [ 0.274127] acpiphp: Slot [3] registered [ 0.275219] acpiphp: Slot [4] registered [ 0.277119] acpiphp: Slot [5] registered [ 0.278127] acpiphp: Slot [6] registered [ 0.280148] acpiphp: Slot [7] registered [ 0.282201] acpiphp: Slot [8] registered [ 0.283148] acpiphp: Slot [9] registered [ 0.285129] acpiphp: Slot [10] registered [ 0.287118] acpiphp: Slot [11] registered [ 0.288118] acpiphp: Slot [12] registered [ 0.290120] acpiphp: Slot [13] registered [ 0.291121] acpiphp: Slot [14] registered [ 0.293133] acpiphp: Slot [15] registered [ 0.294101] acpiphp: Slot [16] registered [ 0.295086] acpiphp: Slot [17] registered [ 0.296157] acpiphp: Slot [18] registered [ 0.298139] acpiphp: Slot [19] registered [ 0.299000] acpiphp: Slot [20] registered [ 0.299000] acpiphp: Slot [21] registered [ 0.299299] acpiphp: Slot [22] registered [ 0.301209] acpiphp: Slot [23] registered [ 0.303175] acpiphp: Slot [24] registered [ 0.305159] acpiphp: Slot [25] registered [ 0.307194] acpiphp: Slot [26] registered [ 0.308214] acpiphp: Slot [27] registered [ 0.310158] acpiphp: Slot [28] registered [ 0.312145] acpiphp: Slot [29] registered [ 0.314159] acpiphp: Slot [30] registered [ 0.316183] acpiphp: Slot [31] registered [ 0.317073] PCI host bridge to bus 0000:00 [ 0.318049] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.322024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.325028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.328040] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.331038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337974] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342670] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351768] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.358011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.365020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369590] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372102] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376517] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.395034] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.401023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437019] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.446000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478022] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.485000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.492022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.498016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.510016] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.519545] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.533017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.550029] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.558000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.562015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.568016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.591000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.594018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.600019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616015] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.627158] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.629468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.632378] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.634443] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.642113] iommu: Default domain type: Passthrough [ 0.643420] SCSI subsystem initialized [ 0.644140] ACPI: bus type USB registered [ 0.645153] usbcore: registered new interface driver usbfs [ 0.647099] usbcore: registered new interface driver hub [ 0.649092] usbcore: registered new device driver usb [ 0.651233] pps_core: LinuxPPS API ver. 1 registered [ 0.653012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.656068] PTP clock support registered [ 0.658321] EDAC MC: Ver: 3.0.0 [ 0.660174] PCI: Using ACPI for IRQ routing [ 0.662920] NetLabel: Initializing [ 0.664012] NetLabel: domain hash size = 128 [ 0.665010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.668096] NetLabel: unlabeled traffic allowed by default [ 0.670124] vgaarb: loaded [ 0.671250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.678281] clocksource: Switched to clocksource kvm-clock [ 0.783178] VFS: Disk quotas dquot_6.6.0 [ 0.784843] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.787444] *** VALIDATE ramfs *** [ 0.788406] *** VALIDATE hugetlbfs *** [ 0.790410] pnp: PnP ACPI init [ 0.792893] pnp: PnP ACPI: found 6 devices [ 0.808988] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812709] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.815365] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.817451] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.820101] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.822751] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.825478] NET: Registered protocol family 2 [ 0.827750] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.833708] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.837136] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.842769] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.846546] TCP: Hash tables configured (established 65536 bind 65536) [ 0.850247] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.853595] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856907] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.859897] NET: Registered protocol family 1 [ 0.862363] RPC: Registered named UNIX socket transport module. [ 0.865057] RPC: Registered udp transport module. [ 0.866389] RPC: Registered tcp transport module. [ 0.868166] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870296] NET: Registered protocol family 44 [ 0.871803] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873994] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.876393] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.879017] PCI: CLS 0 bytes, default 64 [ 0.880962] Unpacking initramfs... [ 2.266847] debug: unmapping init [mem 0xffff8b7ebcc54000-0xffff8b7ebffbffff] [ 2.270504] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.272377] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.275452] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.785779] Initialise system trusted keyrings [ 2.787034] Key type blacklist registered [ 2.788710] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.798867] zbud: loaded [ 2.806090] *** VALIDATE nfs *** [ 2.807501] *** VALIDATE nfs4 *** [ 2.809161] pstore: using deflate compression [ 2.812718] Platform Keyring initialized [ 2.913231] NET: Registered protocol family 38 [ 2.914963] Key type asymmetric registered [ 2.916408] Asymmetric key parser 'x509' registered [ 2.918268] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.921854] io scheduler mq-deadline registered [ 2.923877] io scheduler kyber registered [ 2.925524] io scheduler bfq registered [ 2.933423] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.936292] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.938679] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.941642] ACPI: Power Button [PWRF] [ 3.033971] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.125725] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.314528] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.408270] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.593810] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.626745] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.661725] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.666660] Non-volatile memory driver v1.3 [ 3.668527] Linux agpgart interface v0.103 [ 3.700334] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.702828] vda: detected capacity change from 0 to 68521984 [ 3.717745] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.720150] vdb: detected capacity change from 0 to 1073741824 [ 3.732173] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.734997] vdc: detected capacity change from 0 to 2621440000 [ 3.752186] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.754884] vdd: detected capacity change from 0 to 2621440000 [ 3.769800] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.773192] vde: detected capacity change from 0 to 4294967296 [ 3.786048] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.788728] vdf: detected capacity change from 0 to 4294967296 [ 3.794232] libphy: Fixed MDIO Bus: probed [ 3.802303] usbcore: registered new interface driver usbserial_generic [ 3.804854] usbserial: USB Serial support registered for generic [ 3.807138] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.811103] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.813071] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.815408] mousedev: PS/2 mouse device common for all mice [ 3.818387] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.821787] rtc_cmos 00:05: RTC can wake from S4 [ 3.828516] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.834159] rtc_cmos 00:05: registered as rtc0 [ 3.837493] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.841374] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.843104] intel_pstate: CPU model not supported [ 3.848332] hid: raw HID events driver (C) Jiri Kosina [ 3.852794] usbcore: registered new interface driver usbhid [ 3.854935] usbhid: USB HID core driver [ 3.856526] drop_monitor: Initializing network drop monitor service [ 3.858882] Initializing XFRM netlink socket [ 3.860802] NET: Registered protocol family 10 [ 3.863687] Segment Routing with IPv6 [ 3.865094] NET: Registered protocol family 17 [ 3.868621] mpls_gso: MPLS GSO support [ 3.874691] RAS: Correctable Errors collector initialized. [ 3.877948] AVX version of gcm_enc/dec engaged. [ 3.879526] AES CTR mode by8 optimization enabled [ 3.968263] sched_clock: Marking stable (3968238978, 0)->(4931754399, -963515421) [ 3.972448] registered taskstats version 1 [ 3.974268] Loading compiled-in X.509 certificates [ 3.976435] zswap: loaded using pool lzo/zbud [ 4.000898] Key type big_key registered [ 4.013421] Key type encrypted registered [ 4.015350] ima: No TPM chip found, activating TPM-bypass! [ 4.017674] ima: Allocated hash algorithm: sha1 [ 4.019338] ima: No architecture policies found [ 4.021302] evm: Initialising EVM extended attributes: [ 4.023169] evm: security.selinux [ 4.024412] evm: security.ima [ 4.025547] evm: security.capability [ 4.026852] evm: HMAC attrs: 0x1 [ 4.032988] rtc_cmos 00:05: setting system clock to 2025-11-16 22:35:50 UTC (1763332550) [ 4.039912] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 4.042982] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 4.052141] Write protecting the kernel read-only data: 28672k [ 4.055601] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 4.058525] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 4.095482] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.103684] systemd[1]: Detected virtualization kvm. [ 4.105583] systemd[1]: Detected architecture x86-64. [ 4.107542] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.131523] systemd[1]: No hostname configured. [ 4.133357] systemd[1]: Set hostname to . [ 4.135551] random: systemd: uninitialized urandom read (16 bytes read) [ 4.138160] systemd[1]: Initializing machine ID from random generator. [ 4.188982] random: ln: uninitialized urandom read (6 bytes read) [ 4.291420] random: systemd: uninitialized urandom read (16 bytes read) [ 4.294146] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.298658] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.302399] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.997689] device-mapper: uevent: version 1.0.3 [ 5.000832] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.763328] virtio_net virtio0 ens2: renamed from eth0 [ 5.801880] random: fast init done [ 5.859068] scsi host0: ata_piix [ 5.865822] scsi host1: ata_piix [ 5.867616] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.870026] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.552676] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.534971] random: crng init done [ 10.536429] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.014484] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.234136] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.519175] SELinux: Disabled at runtime. [ 12.576311] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.585810] systemd[1]: Detected virtualization kvm. [ 12.587778] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.178811] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.182489] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.186887] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.191121] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.196298] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.204067] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.210282] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ 13.304894] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.637393] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.032259] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.087921] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.223413] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.233344] EDAC sbridge: Ver: 1.1.2 [ 17.758859] Key type dns_resolver registered [ 18.249319] NFS: Registering the id_resolver key type [ 18.251991] Key type id_resolver registered [ 18.254859] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg429-server login: [ 43.152109] hrtimer: interrupt took 3860964 ns [ 77.250954] libcfs: loading out-of-tree module taints kernel. [ 77.281388] Key type ._llcrypt registered [ 77.283035] Key type .llcrypt registered [ 77.373442] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 94.515128] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 96.294454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.304478] alg: No test for adler32 (adler32-zlib) [ 97.763362] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 98.563049] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 100.297799] Key type lgssc registered [ 101.857765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 121.743230] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.895098] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.983447] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.271810] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.347534] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 158.448817] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 175.545088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 175.624360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.645314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.961477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 177.031808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 177.172695] Lustre: lustre-MDT0000: new disk, initializing [ 177.304146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.334754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 181.394094] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.866846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.950347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 195.002696] Lustre: 6513:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.037614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.046124] Lustre: Skipped 1 previous similar message [ 195.110514] Lustre: lustre-MDT0001: new disk, initializing [ 195.170748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 195.192229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 195.204990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.634188] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.794102] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 211.833916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.895101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 212.141417] Lustre: lustre-OST0000: new disk, initializing [ 212.145222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.223789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.198358] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.710461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 218.725948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 218.833785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.138835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 231.221775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 231.312580] Lustre: lustre-OST0001: new disk, initializing [ 231.315504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 231.382186] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.558038] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.095627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.103409] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.156440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.528363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.514960] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.395720] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.277960] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing yml_node [ 274.571855] Lustre: DEBUG MARKER: Client: 2.16.61.51 [ 277.330233] Lustre: DEBUG MARKER: MDS: 2.16.61.51 [ 279.933734] Lustre: DEBUG MARKER: OSS: 2.16.61.51 [ 281.359738] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 17:40:26 EST 2025 [ 297.207648] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 298.612637] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 314.849625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.850065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.854198] Lustre: Skipped 2 previous similar messages [ 314.865290] Lustre: Skipped 3 previous similar messages [ 319.417881] LustreError: 12217:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 319.558382] Lustre: server umount lustre-MDT0000 complete [ 323.451552] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763332869 with bad export cookie 790139238570485150 [ 323.454484] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.462037] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 325.088329] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.092439] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.104660] Lustre: Skipped 1 previous similar message [ 325.108984] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.141706] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 328.098905] LustreError: 7793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.099734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.109872] LustreError: 7793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 328.112685] Lustre: Skipped 2 previous similar messages [ 329.640091] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 329.649932] LustreError: 12418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 329.826343] Lustre: server umount lustre-MDT0001 complete [ 333.199141] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 333.202250] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 333.269972] Lustre: server umount lustre-OST0000 complete [ 336.667307] LustreError: 12822:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 336.674278] LustreError: 12822:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 336.794766] Lustre: server umount lustre-OST0001 complete [ 342.634751] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 349.878844] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 358.273712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 364.038532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 369.859768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 376.181745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 383.591983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 383.671568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.044602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.104820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.188772] Lustre: lustre-MDT0000: new disk, initializing [ 384.280559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.299826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.730487] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 399.277809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 399.394425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 399.476077] Lustre: 15007:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.524399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.527038] Lustre: Skipped 1 previous similar message [ 399.590909] Lustre: lustre-MDT0001: new disk, initializing [ 399.707773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.747435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.777270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 403.555031] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 411.560201] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 415.830889] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 424.101702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 424.177543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 424.544575] Lustre: lustre-OST0000: new disk, initializing [ 424.548539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.684326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.535764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.550385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.693364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.202923] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 439.686847] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 444.150488] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.364688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.188633] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.374500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.582739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.588725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.597310] LustreError: Skipped 1 previous similar message [ 452.611909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.614024] Lustre: Skipped 2 previous similar messages [ 457.699091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.702823] Lustre: Skipped 1 previous similar message [ 464.352613] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.387979] LustreError: 17948:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 464.400211] LustreError: 17948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 464.537058] Lustre: server umount lustre-OST0000 complete [ 467.940261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.945733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.950647] Lustre: Skipped 1 previous similar message [ 467.962092] Lustre: Skipped 4 previous similar messages [ 482.271480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 482.369751] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 482.385931] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 482.495887] Lustre: server umount lustre-MDT0000 complete [ 483.304073] LustreError: 15018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.316760] LustreError: 15018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 486.321778] LustreError: 15000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333032 with bad export cookie 790139238570486683 [ 486.337334] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.639292] Lustre: server umount lustre-MDT0001 complete [ 489.834873] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:43:55 (1763333035) [ 495.725788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.365817] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 496.485701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.601926] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 501.738502] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.597927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 511.981393] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 517.635912] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 520.653705] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 528.038025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 536.730268] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 543.796608] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 551.392236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 551.395302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.424489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 551.428399] Lustre: Skipped 5 previous similar messages [ 555.718178] LustreError: 21123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 555.739601] LustreError: 21123:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 555.900814] Lustre: server umount lustre-OST0000 complete [ 560.098365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 560.108108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.123296] Lustre: Skipped 1 previous similar message [ 566.064643] Lustre: server umount lustre-MDT0000 complete [ 566.752105] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.771525] LustreError: 18879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 569.459434] LustreError: 18859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333115 with bad export cookie 790139238570487565 [ 569.465344] LustreError: 18859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 569.466649] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.862371] Lustre: server umount lustre-MDT0001 complete [ 579.655927] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 582.590101] Key type lgssc unregistered [ 582.940517] LNet: 22060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.952425] LNetError: 22060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.978320] LNet: Removed LNI 192.168.204.129@tcp [ 583.902142] Key type .llcrypt unregistered [ 583.904964] Key type ._llcrypt unregistered [ 606.018211] Key type ._llcrypt registered [ 606.019959] Key type .llcrypt registered [ 606.127538] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 607.257603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 607.270434] alg: No test for adler32 (adler32-zlib) [ 608.333343] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 608.563939] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 610.255235] Key type lgssc registered [ 611.613798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 620.140301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.157931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.736767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.842883] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 632.293686] LustreError: 23455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 632.304268] LustreError: 23455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 634.037869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 634.440859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 637.806703] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 642.582923] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 646.247247] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 652.342600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 652.628158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.816569] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 663.287339] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 667.023947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 677.349098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.357174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 677.370452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 677.390927] Lustre: Skipped 1 previous similar message [ 679.056891] LustreError: 26145:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 679.242992] Lustre: server umount lustre-OST0000 complete [ 685.536249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 685.539311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.572518] Lustre: Skipped 1 previous similar message [ 685.585472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.585510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.592757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 687.605904] Lustre: Skipped 1 previous similar message [ 689.605078] LustreError: 26347:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 689.613369] LustreError: 26347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 689.798306] Lustre: server umount lustre-MDT0000 complete [ 692.723872] LustreError: 25626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.745846] LustreError: 25626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 693.466033] LustreError: 23436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333239 with bad export cookie 18202727668140853052 [ 693.477757] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.481332] LustreError: 23436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 693.636619] LustreError: 26548:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 693.660225] LustreError: 26548:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 693.844024] Lustre: server umount lustre-MDT0001 complete [ 703.848933] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 707.000923] Key type lgssc unregistered [ 707.322300] LNet: 27080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.327451] LNetError: 27080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.346939] LNet: Removed LNI 192.168.204.129@tcp [ 708.414172] Key type .llcrypt unregistered [ 708.415737] Key type ._llcrypt unregistered [ 733.866421] Key type ._llcrypt registered [ 733.868322] Key type .llcrypt registered [ 734.074977] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 735.026639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.064863] alg: No test for adler32 (adler32-zlib) [ 736.284838] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 736.525683] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 738.183189] Key type lgssc registered [ 739.453474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 747.760414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 747.791709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.318075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 753.524931] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 758.757119] LustreError: 28475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.769855] LustreError: 28475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 761.940685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 762.424358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 766.211191] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 770.801658] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 773.866484] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 784.583669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 784.877168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.938745] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 794.132386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 794.936695] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 802.273074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.286317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.310630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.323715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 804.324136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 804.326591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.457171] LustreError: 31065:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 806.558190] Lustre: server umount lustre-OST0000 complete [ 813.538709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 813.549981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.579725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.582524] Lustre: Skipped 1 previous similar message [ 819.682197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 819.684902] Lustre: Skipped 3 previous similar messages [ 824.287302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 824.347598] LustreError: 31266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 824.350272] LustreError: 31266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 824.397808] Lustre: server umount lustre-MDT0000 complete [ 824.804618] LustreError: 29189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.819790] LustreError: 29189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 827.427193] LustreError: 28456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333373 with bad export cookie 16914222525435022463 [ 827.437719] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.598951] LustreError: 31469:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 827.612144] LustreError: 31469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 827.777301] Lustre: server umount lustre-MDT0001 complete [ 835.509476] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 838.000849] Key type lgssc unregistered [ 838.235538] LNet: 32002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.247837] LNetError: 32002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.259875] LNet: Removed LNI 192.168.204.129@tcp [ 839.011275] Key type .llcrypt unregistered [ 839.013129] Key type ._llcrypt unregistered [ 857.723211] Key type ._llcrypt registered [ 857.724617] Key type .llcrypt registered [ 857.790622] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 858.792880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.919730] alg: No test for adler32 (adler32-zlib) [ 859.931298] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 860.167304] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 861.815170] Key type lgssc registered [ 862.726549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 869.361503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 869.387340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.938499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 874.751637] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 879.008331] LustreError: 33376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.022717] LustreError: 33376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 881.062724] LustreError: 33376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.581496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 882.980773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 886.832386] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 892.296839] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 895.971583] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 902.035887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 902.366099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.645736] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 912.426934] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 915.693629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 924.128800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.132493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.143462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.180923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.181635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.185174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.183123] LustreError: 35639:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 930.283365] Lustre: server umount lustre-OST0000 complete [ 934.368523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 934.372052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.384934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.387520] Lustre: Skipped 1 previous similar message [ 936.418578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.419192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.240329] LustreError: 35841:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 939.257986] LustreError: 35841:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 939.309123] Lustre: server umount lustre-MDT0000 complete [ 941.537443] LustreError: 33376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.560898] LustreError: 33376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 942.199960] LustreError: 33356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333488 with bad export cookie 10856055216402064756 [ 942.210503] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 942.211143] LustreError: 33356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 942.353968] LustreError: 36045:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 942.356637] LustreError: 36045:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 942.492727] Lustre: server umount lustre-MDT0001 complete [ 950.439834] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 952.821422] Key type lgssc unregistered [ 953.110807] LNet: 36575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.126152] LNetError: 36575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.146485] LNet: Removed LNI 192.168.204.129@tcp [ 953.924559] Key type .llcrypt unregistered [ 953.927368] Key type ._llcrypt unregistered [ 973.168858] Key type ._llcrypt registered [ 973.170291] Key type .llcrypt registered [ 973.271441] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 974.224747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.243343] alg: No test for adler32 (adler32-zlib) [ 975.331859] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 975.513839] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 977.191253] Key type lgssc registered [ 978.146888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 984.961398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 984.978271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.470250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.040690] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 995.809974] LustreError: 37947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.835331] LustreError: 37947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 997.828833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 998.201255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1001.963505] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1008.041197] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.248656] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1017.419996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1017.688245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.268384] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1028.919173] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1031.099275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1039.760798] Lustre: lustre-OST0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 1041.378721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.395139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.405144] Lustre: Skipped 1 previous similar message [ 1041.409921] Lustre: Skipped 1 previous similar message [ 1044.542827] LustreError: 40195:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1044.687914] Lustre: server umount lustre-OST0000 complete [ 1064.929321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1064.941263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.952183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.952713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.837997] LustreError: 40572:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1069.850259] LustreError: 40572:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1069.934856] Lustre: server umount lustre-MDT0000 complete [ 1071.075093] LustreError: 37948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.091062] LustreError: 37948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1072.729069] LustreError: 37928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333619 with bad export cookie 858351875949001389 [ 1072.731080] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.743052] LustreError: 37928:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1078.956075] LustreError: 40775:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1078.963765] LustreError: 40775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1079.132860] Lustre: server umount lustre-MDT0001 complete [ 1088.888675] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1091.431339] Key type lgssc unregistered [ 1091.705255] LNet: 41308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.717025] LNetError: 41308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.727860] LNet: Removed LNI 192.168.204.129@tcp [ 1092.456940] Key type .llcrypt unregistered [ 1092.459526] Key type ._llcrypt unregistered [ 1114.263102] Key type ._llcrypt registered [ 1114.264829] Key type .llcrypt registered [ 1114.393234] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1115.791021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1115.815742] alg: No test for adler32 (adler32-zlib) [ 1116.933176] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1117.186704] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1118.863192] Key type lgssc registered [ 1120.042674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.353759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.370525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.867442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.568233] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1138.147829] LustreError: 42700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.154664] LustreError: 42700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1139.778842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1140.129211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.390606] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1148.341649] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.722469] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1157.469111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1157.821730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.742036] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1169.846897] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1173.175851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1182.085812] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (stopping) [ 1183.505271] LustreError: 44948:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1183.632442] Lustre: server umount lustre-MDT0000 complete [ 1183.713610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.720815] LustreError: 42697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.727687] LustreError: 42697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1187.199148] LustreError: 42695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.231511] LustreError: 42695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1187.525215] LustreError: 44949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763333733 with bad export cookie 10964043845182538143 [ 1187.542155] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.546210] LustreError: 44949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1188.833158] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1188.838709] Lustre: Skipped 2 previous similar messages [ 1188.857606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1192.304153] LustreError: 43912:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1192.328355] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.29@tcp (stopping) [ 1192.331643] LustreError: 43912:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1193.775479] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1193.778405] LustreError: 45150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1194.057563] Lustre: server umount lustre-MDT0001 complete [ 1215.049167] LustreError: 45375:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1215.052235] LustreError: 45375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1215.161882] Lustre: server umount lustre-OST0000 complete [ 1228.354780] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1230.606376] Key type lgssc unregistered [ 1230.850942] LNet: 46213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.859297] LNetError: 46213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.874133] LNet: Removed LNI 192.168.204.129@tcp [ 1231.659528] Key type .llcrypt unregistered [ 1231.665215] Key type ._llcrypt unregistered [ 1250.040034] Key type ._llcrypt registered [ 1250.041516] Key type .llcrypt registered [ 1250.143596] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1250.885320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1250.930176] alg: No test for adler32 (adler32-zlib) [ 1251.912418] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1252.075419] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1253.703184] Key type lgssc registered [ 1254.633745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.805896] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.824085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1276.447653] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1281.567331] LustreError: 47560:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 1308.255511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.167690] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1320.334513] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.836172] LustreError: 48268:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1484.989438] Lustre: server umount lustre-OST0000 complete [ 1500.199606] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1503.146423] Key type lgssc unregistered [ 1503.417059] LNet: 49105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.431052] LNetError: 49105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.492680] LNet: Removed LNI 192.168.204.129@tcp [ 1505.621677] Key type .llcrypt unregistered [ 1505.626760] Key type ._llcrypt unregistered [ 1532.394670] Key type ._llcrypt registered [ 1532.396085] Key type .llcrypt registered [ 1532.510559] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1533.313518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.398116] alg: No test for adler32 (adler32-zlib) [ 1534.399861] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1534.633404] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1536.279185] Key type lgssc registered [ 1537.267280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1544.422867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1544.442941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.933576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1549.156398] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1556.206621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1556.529255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1559.461435] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1564.203680] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.792870] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.624444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.944597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.371548] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1584.502156] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1588.800268] LustreError: 52758:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1588.883707] Lustre: server umount lustre-OST0000 complete [ 1595.873747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.878711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.879585] Lustre: Skipped 1 previous similar message [ 1595.887707] Lustre: Skipped 1 previous similar message [ 1598.508162] LustreError: 52958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1598.516252] LustreError: 52958:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1598.592857] Lustre: server umount lustre-MDT0000 complete [ 1600.995941] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.003506] LustreError: 50514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1601.966026] LustreError: 50493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334148 with bad export cookie 16077912722516588870 [ 1601.991849] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1612.268216] LustreError: 53161:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1612.281590] LustreError: 53161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1612.465415] Lustre: server umount lustre-MDT0001 complete [ 1621.531849] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1623.953606] Key type lgssc unregistered [ 1624.205038] LNet: 53694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.213666] LNetError: 53694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1624.235930] LNet: Removed LNI 192.168.204.129@tcp [ 1625.001105] Key type .llcrypt unregistered [ 1625.002918] Key type ._llcrypt unregistered [ 1644.601594] Key type ._llcrypt registered [ 1644.602890] Key type .llcrypt registered [ 1644.677956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1645.176329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.204924] alg: No test for adler32 (adler32-zlib) [ 1646.238150] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1646.378740] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1648.015240] Key type lgssc registered [ 1648.888562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.550495] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1654.562368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1670.111747] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1675.231340] LustreError: 55039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 1701.919257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.333543] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1711.837581] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.921935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.327334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1718.428111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1721.719418] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1723.361145] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.389383] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1728.483172] LustreError: 55065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.293857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1729.586039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1732.837504] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1738.135389] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1741.425415] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1743.841989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1743.847202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.859083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.868236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.878844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.889227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.915472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1746.919498] Lustre: Skipped 1 previous similar message [ 1752.033661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.038488] Lustre: Skipped 1 previous similar message [ 1757.151964] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1757.153620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.160482] Lustre: Skipped 1 previous similar message [ 1757.176107] LustreError: 57316:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1757.223609] Lustre: server umount lustre-OST0000 complete [ 1777.633122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1777.655964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1782.702705] LustreError: 57696:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1782.720434] LustreError: 57696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1782.757989] LustreError: 55968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.903762] Lustre: server umount lustre-MDT0000 complete [ 1786.097837] LustreError: 55047:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334332 with bad export cookie 12814289882726153803 [ 1786.104191] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.112511] LustreError: 55047:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1792.334103] LustreError: 57897:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1792.336562] LustreError: 57897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1792.569060] Lustre: server umount lustre-MDT0001 complete [ 1803.267718] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1805.854307] Key type lgssc unregistered [ 1806.140834] LNet: 58430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.152483] LNetError: 58430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.214264] LNet: Removed LNI 192.168.204.129@tcp [ 1808.161034] Key type .llcrypt unregistered [ 1808.168981] Key type ._llcrypt unregistered [ 1829.503901] Key type ._llcrypt registered [ 1829.505608] Key type .llcrypt registered [ 1829.600534] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1830.327670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1830.411061] alg: No test for adler32 (adler32-zlib) [ 1831.441400] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1831.640673] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1833.308323] Key type lgssc registered [ 1834.239811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1840.716530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1840.731304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.104596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.215500] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1852.387928] LustreError: 59830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1852.410283] LustreError: 59830:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1853.224964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1853.480146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1856.931547] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1862.173461] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.415255] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1871.629981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1871.909339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.030861] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1881.992650] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1887.354382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1897.954299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.960964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1897.961673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1897.963805] Lustre: Skipped 1 previous similar message [ 1897.983197] Lustre: Skipped 1 previous similar message [ 1903.076689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1903.086216] Lustre: Skipped 1 previous similar message [ 1904.004643] LustreError: 62095:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1904.132215] Lustre: server umount lustre-OST0000 complete [ 1908.194432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.200583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.203912] Lustre: Skipped 1 previous similar message [ 1913.314335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.333556] Lustre: Skipped 2 previous similar messages [ 1913.346334] LustreError: 62297:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1913.358579] LustreError: 62297:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1913.443964] Lustre: server umount lustre-MDT0000 complete [ 1916.050142] LustreError: 61448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334462 with bad export cookie 16356356449265965954 [ 1916.052909] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.057037] LustreError: 61448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.140936] LustreError: 62500:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1916.145683] LustreError: 62500:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1916.246527] Lustre: server umount lustre-MDT0001 complete [ 1924.171220] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 1926.630475] Key type lgssc unregistered [ 1926.917960] LNet: 63032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.932180] LNetError: 63032:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.975828] LNet: Removed LNI 192.168.204.129@tcp [ 1928.673993] Key type .llcrypt unregistered [ 1928.676628] Key type ._llcrypt unregistered [ 1953.819897] Key type ._llcrypt registered [ 1953.822251] Key type .llcrypt registered [ 1953.916553] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 1954.606744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.687894] alg: No test for adler32 (adler32-zlib) [ 1955.702490] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1955.897891] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 1957.575155] Key type lgssc registered [ 1958.480937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.362189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.376542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.691201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1967.388587] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1973.711151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1974.060806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.953808] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 1981.216714] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1984.188167] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.594558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1989.975263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.037384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1994.815189] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2000.802903] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2006.993674] Lustre: Failing over lustre-MDT0000 [ 2007.156909] LustreError: 66939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2007.247522] Lustre: server umount lustre-MDT0000 complete [ 2008.480153] LustreError: 64720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2008.506161] LustreError: 64720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2009.572115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.574840] LustreError: 64696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.592440] Lustre: Skipped 2 previous similar messages [ 2009.619975] LustreError: 64696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2013.557848] LustreError: 64720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.358968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.535935] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.707973] Lustre: *** cfs_fail_loc=135, val=0*** [ 2015.817907] LustreError: 67448:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 2015.821306] LustreError: 67448:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -2 [ 2015.824260] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2015.824260] [ 2015.827838] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2015.832943] LustreError: 67436:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2015.838958] LustreError: 67436:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2015.848445] LustreError: 67436:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2015.867734] Lustre: server umount lustre-MDT0000 complete [ 2015.870348] LustreError: 67436:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2018.688214] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.719631] LustreError: 64691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2021.644690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.794813] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.068808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.127945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2023.796254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2025.520443] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2027.510258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2027.526991] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2027.555200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2032.611869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.614370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.616775] Lustre: Skipped 1 previous similar message [ 2032.647493] Lustre: Skipped 1 previous similar message [ 2033.996710] LustreError: 68258:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2034.011601] LustreError: 68258:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2034.190486] Lustre: server umount lustre-OST0000 complete [ 2037.728244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.732279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.746403] Lustre: Skipped 1 previous similar message [ 2037.759521] Lustre: Skipped 1 previous similar message [ 2042.848375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.867292] Lustre: Skipped 1 previous similar message [ 2043.855209] LustreError: 68460:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2043.865815] LustreError: 68460:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2044.043095] Lustre: server umount lustre-MDT0000 complete [ 2047.543601] LustreError: 64676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334594 with bad export cookie 6598130052873539862 [ 2047.544613] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.559528] LustreError: 64676:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.677924] LustreError: 68662:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2047.680733] LustreError: 68662:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2047.933674] Lustre: server umount lustre-MDT0001 complete [ 2057.487082] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2060.372761] Key type lgssc unregistered [ 2060.676637] LNet: 69195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.693615] LNetError: 69195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.712272] LNet: Removed LNI 192.168.204.129@tcp [ 2061.571933] Key type .llcrypt unregistered [ 2061.573483] Key type ._llcrypt unregistered [ 2081.419665] Key type ._llcrypt registered [ 2081.424458] Key type .llcrypt registered [ 2081.496852] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2082.274188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.352497] alg: No test for adler32 (adler32-zlib) [ 2083.439334] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2083.762725] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2085.455479] Key type lgssc registered [ 2086.500734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.535193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.565662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.162959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.525106] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2105.317093] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.325995] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2105.575300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2105.871832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.118273] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2113.765572] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2117.470663] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2123.839719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2124.132102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.237584] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2134.003807] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2140.538262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2150.368846] Lustre: Failing over lustre-MDT0000 [ 2150.525233] LustreError: 72835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2150.640688] Lustre: server umount lustre-MDT0000 complete [ 2150.819389] LustreError: 70560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.895261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.897385] LustreError: 70560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.932820] LustreError: 70560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2160.561659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.747271] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.980808] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2161.019085] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.29@tcp (not set up) [ 2161.175403] LustreError: 73344:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2161.184687] LustreError: 73344:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -9 [ 2161.196836] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2161.196836] [ 2161.202317] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2161.226088] LustreError: 73332:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2161.242883] LustreError: 73332:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2161.262160] LustreError: 73332:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2161.269329] LustreError: 70546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334707 with bad export cookie 16889548362599105689 [ 2161.310951] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.335411] Lustre: server umount lustre-MDT0000 complete [ 2161.337517] LustreError: 73332:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2166.160719] LustreError: 70562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.180203] LustreError: 70562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2168.600977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.719273] LustreError: 73692:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2170.735901] LustreError: 73692:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000060195698 x1848990373015680/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1763334717 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2170.779844] LustreError: 73692:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2171.252274] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.276844] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2171.368034] LustreError: 69875:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006a7e3ede x1848990373017472/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2171.652224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.704759] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2175.524750] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2176.374989] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2177.000963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2177.031502] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2177.095629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2182.539090] LustreError: 74156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2182.549052] LustreError: 74156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2182.625135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.628553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.640747] Lustre: Skipped 2 previous similar messages [ 2182.644189] LustreError: 72166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.659448] LustreError: 72166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2182.735552] Lustre: server umount lustre-OST0000 complete [ 2186.267504] LustreError: 74356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2186.276118] LustreError: 74356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2186.344595] Lustre: server umount lustre-MDT0000 complete [ 2187.233377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2187.234659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.236864] LustreError: Skipped 1 previous similar message [ 2187.259982] Lustre: Skipped 2 previous similar messages [ 2190.139342] LustreError: 70547:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334736 with bad export cookie 16889548362599105871 [ 2190.152941] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2190.356484] LustreError: 74557:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2190.360663] LustreError: 74557:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2190.533852] Lustre: server umount lustre-MDT0001 complete [ 2199.679175] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2202.199485] Key type lgssc unregistered [ 2202.466661] LNet: 75089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.472642] LNetError: 75089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.482876] LNet: Removed LNI 192.168.204.129@tcp [ 2203.303815] Key type .llcrypt unregistered [ 2203.306040] Key type ._llcrypt unregistered [ 2215.299846] Key type ._llcrypt registered [ 2215.304229] Key type .llcrypt registered [ 2215.405155] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2215.612085] Key type .llcrypt unregistered [ 2215.613527] Key type ._llcrypt unregistered [ 2237.129579] Key type ._llcrypt registered [ 2237.131413] Key type .llcrypt registered [ 2237.225322] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2238.132955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.149762] alg: No test for adler32 (adler32-zlib) [ 2239.176630] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2239.363170] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2241.023219] Key type lgssc registered [ 2241.929215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2248.606096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2248.624332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.083752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.265355] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2260.449818] LustreError: 76944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.460828] LustreError: 76944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2260.696713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2261.058396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2264.075132] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2268.852855] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2272.526851] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.046159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2279.322178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.291914] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2290.678723] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.724578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2305.000777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.009211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.029859] Lustre: Skipped 1 previous similar message [ 2305.052571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.127256] LustreError: 79212:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2309.324852] Lustre: server umount lustre-OST0000 complete [ 2315.232420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.240889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.242325] Lustre: Skipped 1 previous similar message [ 2315.246207] Lustre: Skipped 1 previous similar message [ 2318.827157] LustreError: 79413:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2318.829764] LustreError: 79413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2318.912319] Lustre: server umount lustre-MDT0000 complete [ 2320.364127] LustreError: 76938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.393985] LustreError: 76938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2322.254566] LustreError: 76924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334868 with bad export cookie 5774793155036577287 [ 2322.264842] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.269642] LustreError: 76924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2322.433748] LustreError: 79614:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2322.437324] LustreError: 79614:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2322.642487] Lustre: server umount lustre-MDT0001 complete [ 2332.971840] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2335.449840] Key type lgssc unregistered [ 2335.705584] LNet: 80147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.720227] LNetError: 80147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.738688] LNet: Removed LNI 192.168.204.129@tcp [ 2336.632795] Key type .llcrypt unregistered [ 2336.635412] Key type ._llcrypt unregistered [ 2358.751887] Key type ._llcrypt registered [ 2358.753454] Key type .llcrypt registered [ 2358.847976] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2360.222449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2360.231556] alg: No test for adler32 (adler32-zlib) [ 2361.274584] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2361.509613] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2363.239182] Key type lgssc registered [ 2364.525542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.224890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.253196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.055885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.357184] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2384.352770] LustreError: 81540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2384.372445] LustreError: 81540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2387.197819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2387.599695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.096175] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2396.382669] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2399.667850] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2406.212573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2406.459619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.460170] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2416.706480] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2418.742628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2428.899153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.936287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2433.793601] LustreError: 83785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2433.947479] Lustre: server umount lustre-OST0000 complete [ 2438.627934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2438.642293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.651039] Lustre: Skipped 1 previous similar message [ 2438.664886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.673510] Lustre: Skipped 1 previous similar message [ 2443.695820] LustreError: 83988:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2443.706503] LustreError: 83988:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2443.805175] Lustre: server umount lustre-MDT0000 complete [ 2444.284418] LustreError: 81541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.129722] LustreError: 81522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763334993 with bad export cookie 808827461802546882 [ 2447.135120] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.317162] LustreError: 84191:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2447.320246] LustreError: 84191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2447.574944] Lustre: server umount lustre-MDT0001 complete [ 2457.866631] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2460.516818] Key type lgssc unregistered [ 2460.796534] LNet: 84723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.801978] LNetError: 84723:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.865203] LNet: Removed LNI 192.168.204.129@tcp [ 2462.611037] Key type .llcrypt unregistered [ 2462.612841] Key type ._llcrypt unregistered [ 2482.710521] Key type ._llcrypt registered [ 2482.713450] Key type .llcrypt registered [ 2482.854685] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2483.654995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.729992] alg: No test for adler32 (adler32-zlib) [ 2484.797983] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2484.990572] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2486.639203] Key type lgssc registered [ 2487.734798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2496.002739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2496.021904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.632239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.613370] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2507.763919] LustreError: 86108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.796086] LustreError: 86108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2510.864237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2511.201160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2515.721158] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2521.626929] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2524.796554] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2531.763775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2532.054407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.223349] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2542.934301] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2543.364467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2553.825655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.861195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.437155] LustreError: 88356:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2554.694400] Lustre: server umount lustre-OST0000 complete [ 2558.950241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.960297] Lustre: Skipped 1 previous similar message [ 2558.969224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.981483] Lustre: Skipped 2 previous similar messages [ 2564.064940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.190106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.197947] Lustre: Skipped 2 previous similar messages [ 2573.280473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2573.415107] LustreError: 88557:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2573.437477] LustreError: 88557:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2573.652297] Lustre: server umount lustre-MDT0000 complete [ 2574.304841] LustreError: 86109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.350146] LustreError: 86109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2577.764531] LustreError: 88270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335124 with bad export cookie 4074072502029116693 [ 2577.765311] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.777243] LustreError: 88270:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2577.959448] LustreError: 88761:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2577.966560] LustreError: 88761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2578.229991] Lustre: server umount lustre-MDT0001 complete [ 2589.871209] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2592.517747] Key type lgssc unregistered [ 2592.750761] LNet: 89292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.761276] LNetError: 89292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2592.777455] LNet: Removed LNI 192.168.204.129@tcp [ 2593.782901] Key type .llcrypt unregistered [ 2593.784607] Key type ._llcrypt unregistered [ 2617.339821] Key type ._llcrypt registered [ 2617.342135] Key type .llcrypt registered [ 2617.466669] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2618.389291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2618.462532] alg: No test for adler32 (adler32-zlib) [ 2619.579185] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2619.811722] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2621.471251] Key type lgssc registered [ 2622.659430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2629.851415] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2629.875526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2645.535386] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2650.655555] LustreError: 90657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 2677.345769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.500766] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2688.133766] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2694.165438] LustreError: 91515:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2694.324662] Lustre: server umount lustre-OST0000 complete [ 2703.287957] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:20:48 (1763335248) [ 2709.713658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.089307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.774351] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2715.124286] LustreError: 92184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.160559] LustreError: 92184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2720.247905] LustreError: 92184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.254249] LustreError: 92184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.657894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2724.051640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.121408] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2733.121931] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2736.489452] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2743.811088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2744.266584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.968535] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2755.377195] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2756.524159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2769.890148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2769.896467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.911343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.941721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.941733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.981986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2775.107243] LustreError: 94593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2775.113041] LustreError: 94593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2775.217149] Lustre: server umount lustre-OST0000 complete [ 2780.131033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2780.135900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.145219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2780.148931] Lustre: Skipped 1 previous similar message [ 2782.179350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2782.181313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.240759] Lustre: Skipped 1 previous similar message [ 2784.579345] LustreError: 94794:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2784.583609] LustreError: 94794:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2784.853649] Lustre: server umount lustre-MDT0000 complete [ 2787.297330] LustreError: 93347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.342396] LustreError: 93347:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2788.718091] LustreError: 94795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335335 with bad export cookie 13376289474895421221 [ 2788.721820] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.751741] LustreError: 94795:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2788.891498] LustreError: 94996:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2788.894247] LustreError: 94996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2789.191951] Lustre: server umount lustre-MDT0001 complete [ 2800.700647] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2803.334601] Key type lgssc unregistered [ 2803.703629] LNet: 95528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2803.719275] LNetError: 95528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2803.741676] LNet: Removed LNI 192.168.204.129@tcp [ 2804.671909] Key type .llcrypt unregistered [ 2804.673769] Key type ._llcrypt unregistered [ 2828.110968] Key type ._llcrypt registered [ 2828.112357] Key type .llcrypt registered [ 2828.235387] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2829.263248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2829.291574] alg: No test for adler32 (adler32-zlib) [ 2830.496413] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2830.759670] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2832.415623] Key type lgssc registered [ 2833.549625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2841.672159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2841.684700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.196333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.044822] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2848.229020] LustreError: 96921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2848.241310] LustreError: 96921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2853.348070] LustreError: 96921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.018184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2855.344439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.890874] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2863.243786] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.228537] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2872.253988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2872.565661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.238537] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 2882.483816] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2887.921443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2895.328445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.346253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.372320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.404471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.405552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2898.407772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.524473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.535980] Lustre: Skipped 1 previous similar message [ 2908.127458] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.174191] LustreError: 99168:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2908.263661] Lustre: server umount lustre-OST0000 complete [ 2913.760581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.775877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.782152] Lustre: Skipped 2 previous similar messages [ 2918.251898] LustreError: 99370:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2918.254662] LustreError: 99370:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2918.457393] Lustre: server umount lustre-MDT0000 complete [ 2918.884325] LustreError: 96918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.823774] LustreError: 96901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335468 with bad export cookie 12725555229748565698 [ 2921.828243] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2921.835736] LustreError: 96901:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2922.009881] LustreError: 99572:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2922.021133] LustreError: 99572:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2922.169146] Lustre: server umount lustre-MDT0001 complete [ 2931.398169] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 2933.661295] Key type lgssc unregistered [ 2933.918765] LNet: 100104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.926550] LNetError: 100104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.948962] LNet: Removed LNI 192.168.204.129@tcp [ 2934.666599] Key type .llcrypt unregistered [ 2934.668774] Key type ._llcrypt unregistered [ 2950.748763] Key type ._llcrypt registered [ 2950.751310] Key type .llcrypt registered [ 2950.874659] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 2951.677721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2951.832565] alg: No test for adler32 (adler32-zlib) [ 2952.875161] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2953.070950] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 2954.727198] Key type lgssc registered [ 2955.691519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2962.153605] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2962.168301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2977.695485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2982.818987] LustreError: 101328:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 3009.503383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.437361] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3019.485655] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3026.090323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.491205] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3026.498734] LustreError: 102240:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3026.505666] LustreError: 102240:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3026.511856] LustreError: 102240:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3026.518040] LustreError: 101337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335572 with bad export cookie 12311782115804244122 [ 3026.532805] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.627829] Lustre: server umount lustre-MDT0000 complete [ 3026.704261] LustreError: 102240:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3052.516221] LustreError: 102670:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3052.648275] Lustre: server umount lustre-OST0000 complete [ 3060.108408] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3067.329458] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3075.462667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3081.198949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3087.016402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3092.795455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.502914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3099.561854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.810804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3099.863087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3099.992534] Lustre: lustre-MDT0000: new disk, initializing [ 3100.090973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.123206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3103.773551] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3113.770139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.860678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3113.958870] Lustre: 105012:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3114.026786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3114.030384] Lustre: Skipped 1 previous similar message [ 3114.120758] Lustre: lustre-MDT0001: new disk, initializing [ 3114.242719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3114.278899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3114.286411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3118.106672] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3125.624519] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3129.517115] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3135.468283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3135.563732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3135.872355] Lustre: lustre-OST0000: new disk, initializing [ 3135.880753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3135.973562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.726850] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3142.210870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3142.226356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3142.359878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3147.144618] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3151.133129] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3151.418920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3154.694638] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3154.867608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3157.985373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3157.990624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.991743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3158.000541] LustreError: Skipped 1 previous similar message [ 3158.018772] Lustre: Skipped 1 previous similar message [ 3162.489945] LustreError: 107950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3162.504543] LustreError: 107950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3162.642614] Lustre: server umount lustre-OST0000 complete [ 3168.224926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.230099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3168.234978] Lustre: Skipped 2 previous similar messages [ 3168.249383] Lustre: Skipped 1 previous similar message [ 3172.043426] LustreError: 108152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3172.062183] LustreError: 108152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3172.164773] Lustre: server umount lustre-MDT0000 complete [ 3175.045844] LustreError: 105003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335721 with bad export cookie 12311782115804244213 [ 3175.047137] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3175.060965] LustreError: 105003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3175.177360] LustreError: 108355:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3175.179528] LustreError: 108355:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3175.365408] Lustre: server umount lustre-MDT0001 complete [ 3182.212394] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:28:47 (1763335727) [ 3185.053791] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3198.459085] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3205.617303] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3213.114839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3217.781964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3222.748441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3229.218826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.984640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3236.091374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.290194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3236.343893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3236.489366] Lustre: lustre-MDT0000: new disk, initializing [ 3236.584412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.626805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3241.445219] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3251.708507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3251.864866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3251.968404] Lustre: 111490:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3252.021645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3252.023872] Lustre: Skipped 1 previous similar message [ 3252.078592] Lustre: lustre-MDT0001: new disk, initializing [ 3252.206141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.247389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3252.257955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3256.309226] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3263.208170] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3265.943550] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3271.917889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3272.008813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3272.206926] Lustre: lustre-OST0000: new disk, initializing [ 3272.212625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3272.318096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.191177] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3280.944358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3280.948886] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3281.025761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3283.979714] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3287.391572] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3287.616242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3291.064678] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3291.252300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3296.226064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3296.229666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.242077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3299.010182] LustreError: 114431:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3299.015478] LustreError: 114431:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3299.138224] Lustre: server umount lustre-OST0000 complete [ 3303.396677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3303.403180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.413434] Lustre: Skipped 1 previous similar message [ 3303.419711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.423438] Lustre: Skipped 1 previous similar message [ 3308.639690] LustreError: 114633:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3308.642938] LustreError: 114633:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3308.748483] Lustre: server umount lustre-MDT0000 complete [ 3312.104060] LustreError: 112734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.125081] LustreError: 112734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3312.243878] LustreError: 111480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335858 with bad export cookie 12311782115804245102 [ 3312.247505] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.262640] LustreError: 111480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3312.627772] Lustre: server umount lustre-MDT0001 complete [ 3320.261545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.932351] LustreError: 115309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.011670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.859476] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3326.434329] LustreError: 115309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.559129] LustreError: 115309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.448674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3337.956758] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3343.270618] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3346.832408] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3353.685780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.203756] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3364.742093] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3375.750663] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3379.680947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3379.688503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3379.709835] Lustre: Skipped 1 previous similar message [ 3379.714116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.719151] Lustre: Skipped 2 previous similar messages [ 3384.706410] LustreError: 117653:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3384.709071] LustreError: 117653:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3384.889990] Lustre: server umount lustre-OST0000 complete [ 3390.432205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3390.461162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.475604] Lustre: Skipped 1 previous similar message [ 3390.486163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3390.489050] Lustre: Skipped 2 previous similar messages [ 3395.005729] Lustre: server umount lustre-MDT0000 complete [ 3398.114892] LustreError: 115308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.123350] LustreError: 115308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3399.142413] LustreError: 115289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763335945 with bad export cookie 12311782115804245984 [ 3399.152097] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3399.165081] LustreError: 115289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3399.637428] Lustre: server umount lustre-MDT0001 complete [ 3411.087685] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 3415.041403] Key type lgssc unregistered [ 3415.470892] LNet: 118591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3415.483698] LNetError: 118591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3415.509940] LNet: Removed LNI 192.168.204.129@tcp [ 3416.569747] Key type .llcrypt unregistered [ 3416.571567] Key type ._llcrypt unregistered [ 3433.868631] Key type ._llcrypt registered [ 3433.871239] Key type .llcrypt registered [ 3434.016893] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 3446.167298] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 3446.918711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.078648] alg: No test for adler32 (adler32-zlib) [ 3448.105872] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 3448.296604] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 3449.943214] Key type lgssc registered [ 3451.033765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3460.198780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3466.488246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.149181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.792211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.600334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3486.655344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3486.680304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.914296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3487.947690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3488.012289] Lustre: lustre-MDT0000: new disk, initializing [ 3488.102312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.119451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.820355] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3501.572928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.655292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3501.767667] Lustre: 121700:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3501.850745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3501.853241] Lustre: Skipped 1 previous similar message [ 3501.968229] Lustre: lustre-MDT0001: new disk, initializing [ 3502.120651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3502.177057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3502.209975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3506.064956] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3514.111214] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3518.475575] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3526.670634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3526.791754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3527.301810] Lustre: lustre-OST0000: new disk, initializing [ 3527.310630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3527.443424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.861249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3532.868844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3532.998164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3533.909803] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3541.719336] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3546.174044] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.391253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3550.314683] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3550.518084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3553.763733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.769468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.774119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.777607] Lustre: Skipped 1 previous similar message [ 3553.798684] Lustre: Skipped 1 previous similar message [ 3558.463457] LustreError: 124642:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3558.685062] Lustre: server umount lustre-OST0000 complete [ 3563.490819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3563.498400] LustreError: Skipped 1 previous similar message [ 3563.510934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3563.529600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.647469] LustreError: 124844:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3568.656693] LustreError: 124844:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3568.748478] Lustre: server umount lustre-MDT0000 complete [ 3572.196804] LustreError: 121694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336118 with bad export cookie 443064190086419322 [ 3572.199127] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.223402] LustreError: 121694:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3572.361453] LustreError: 125046:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3572.366891] LustreError: 125046:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3572.610285] Lustre: server umount lustre-MDT0001 complete [ 3580.428873] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:35:25 (1763336125) [ 3587.341036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.960103] LustreError: 125718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.976310] LustreError: 125718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3588.059072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.430158] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3593.189052] LustreError: 125718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.216691] LustreError: 125718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3598.315988] LustreError: 125718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.296711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3602.896915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.812085] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3614.247324] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3617.882420] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3623.392949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3623.394853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.411012] Lustre: Skipped 2 previous similar messages [ 3623.419106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.422182] Lustre: Skipped 1 previous similar message [ 3627.555769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3627.558160] Lustre: Skipped 1 previous similar message [ 3632.609159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.625253] Lustre: Skipped 1 previous similar message [ 3634.143260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3634.241129] LustreError: 127061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3634.250743] LustreError: 127061:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3634.338980] Lustre: server umount lustre-MDT0000 complete [ 3637.743671] LustreError: 125712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.760884] LustreError: 125712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3638.641214] LustreError: 125699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336185 with bad export cookie 443064190086420204 [ 3638.641708] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.663368] LustreError: 125699:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3638.784814] LustreError: 127268:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3638.795636] LustreError: 127268:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3639.057147] Lustre: server umount lustre-MDT0001 complete [ 3647.608968] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:36:32 (1763336192) [ 3654.445818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3669.919619] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3675.045321] LustreError: 127911:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 3700.831657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3706.785476] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3711.880351] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3713.776608] LustreError: 128576:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3713.783167] LustreError: 128576:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3713.897266] Lustre: server umount lustre-OST0000 complete [ 3721.879524] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:37:47 (1763336267) [ 3728.647922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3729.084086] LustreError: 129247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.167080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.774832] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3738.613657] LustreError: 129246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.654990] LustreError: 129246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3742.703661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3747.766213] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3754.871330] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3759.210355] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3766.559787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3766.832445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.838711] Lustre: Skipped 1 previous similar message [ 3772.480259] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3778.111334] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3789.282558] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.286243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3789.286737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.306804] Lustre: Skipped 1 previous similar message [ 3791.664925] LustreError: 131492:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3791.680160] LustreError: 131492:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3791.750994] Lustre: server umount lustre-MDT0000 complete [ 3794.401348] LustreError: 129246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.417602] LustreError: 129246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3795.416688] LustreError: 129227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336341 with bad export cookie 443064190086420869 [ 3795.422714] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3795.426310] LustreError: 129227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3795.895351] Lustre: server umount lustre-MDT0001 complete [ 3808.887414] Lustre: server umount lustre-OST0000 complete [ 3817.287847] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:39:22 (1763336362) [ 3823.617405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.552592] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3829.219911] LustreError: 132578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3829.244904] LustreError: 132578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3837.174565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3837.661958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.665714] Lustre: Skipped 1 previous similar message [ 3841.885987] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3847.086809] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3851.200572] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3857.977730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3863.852413] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3868.670709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3869.092716] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3872.726539] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3872.890347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3876.141523] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3876.404366] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3878.884185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3878.887463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.900987] LustreError: Skipped 1 previous similar message [ 3878.908693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3878.908701] Lustre: Skipped 2 previous similar messages [ 3878.941455] Lustre: Skipped 2 previous similar messages [ 3884.666101] LustreError: 135322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3884.685410] LustreError: 135322:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3884.835757] Lustre: server umount lustre-OST0000 complete [ 3889.120467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3889.143741] Lustre: Skipped 1 previous similar message [ 3894.760042] Lustre: server umount lustre-MDT0000 complete [ 3898.370528] LustreError: 132557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336444 with bad export cookie 443064190086421933 [ 3898.380105] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.385012] LustreError: 132557:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3898.726790] Lustre: server umount lustre-MDT0001 complete [ 3906.776269] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:40:52 (1763336452) [ 3913.912854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3929.568484] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3934.692583] LustreError: 136370:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 3965.778951] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3970.025186] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3976.111756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.523221] LustreError: 136396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3976.544088] LustreError: 136396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3976.594429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3976.597598] Lustre: Skipped 2 previous similar messages [ 3976.658819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3980.163442] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3987.262524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3991.294122] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 3996.173875] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3999.307433] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4002.680277] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4002.880745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4006.717395] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4006.893887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4009.953957] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.957945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4009.968414] Lustre: Skipped 2 previous similar messages [ 4009.973728] Lustre: Skipped 9 previous similar messages [ 4023.264229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4023.391967] LustreError: 139148:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4023.398640] LustreError: 139148:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4023.450734] Lustre: server umount lustre-MDT0000 complete [ 4026.955248] LustreError: 139149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336573 with bad export cookie 443064190086422962 [ 4026.958558] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.975253] LustreError: 139149:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4027.271093] Lustre: server umount lustre-MDT0001 complete [ 4040.941797] Lustre: server umount lustre-OST0000 complete [ 4048.712549] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:43:14 (1763336594) [ 4054.248779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4069.791646] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4074.911496] LustreError: 140199:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 4105.871474] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4111.344436] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4117.487103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.810726] LustreError: 140226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.832131] LustreError: 140226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4117.944369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4121.249815] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4129.254954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4133.059642] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4138.103322] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.071459] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.677530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4146.778859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.919255] Lustre: lustre-OST0001: new disk, initializing [ 4146.924492] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4151.956550] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4156.456896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4156.480209] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4156.615904] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4159.562609] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4163.901672] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4164.144798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4167.912715] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4168.110547] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4170.722502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.743470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.770287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.773105] Lustre: Skipped 6 previous similar messages [ 4175.740025] LustreError: 143930:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4175.743074] LustreError: 143930:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4175.781285] Lustre: server umount lustre-OST0000 complete [ 4193.247161] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4193.451278] Lustre: server umount lustre-OST0001 complete [ 4203.427550] Lustre: server umount lustre-MDT0000 complete [ 4207.145712] LustreError: 140207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336753 with bad export cookie 443064190086423907 [ 4207.156424] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4207.172142] LustreError: 140207:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4207.436966] Lustre: server umount lustre-MDT0001 complete [ 4212.894697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4216.524759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4219.713311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4223.235127] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4229.862279] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:46:15 (1763336775) [ 4231.224977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4232.968180] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4234.715943] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:46:20 (1763336780) [ 4240.877441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.191209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4241.219723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4241.406978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4241.412323] Lustre: Skipped 5 previous similar messages [ 4245.195687] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4253.394390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4253.570308] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4253.602108] Lustre: 146174:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4257.569602] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4262.571918] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4265.081337] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4270.669617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4270.816937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4276.199866] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4278.348102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4281.589760] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4285.231323] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4285.411190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4289.039730] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4289.240872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4293.604753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.607163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4293.635417] Lustre: Skipped 5 previous similar messages [ 4293.646867] LustreError: Skipped 3 previous similar messages [ 4297.547475] Lustre: server umount lustre-OST0000 complete [ 4317.827943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.987613] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4327.151871] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4332.164045] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4334.515249] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.521819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.522474] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4334.536069] Lustre: Skipped 1 previous similar message [ 4335.636267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4339.711529] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4339.937955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4343.944748] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e0b4cf37800.ost_server_uuid 50 [ 4345.843129] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9e0b4cf37800.ost_server_uuid in FULL state after 0 sec [ 4346.068789] LustreError: 146179:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.29@tcp) returned error from blocking AST (req@00000000a41c7f08 x1848991803487488 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000081253c49/0x62614888ecc2eb6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.29@tcp remote: 0x650796453c54570c expref: 5 pid: 149073 timeout: 4446 lvb_type: 0 [ 4346.130208] LustreError: lustre-MDT0000: A client on nid 192.168.204.29@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4346.164634] LustreError: 146170:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.29@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000081253c49/0x62614888ecc2eb6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.29@tcp remote: 0x650796453c54570c expref: 6 pid: 149073 timeout: 0 lvb_type: 0 [ 4351.974551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4351.995435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4351.998644] Lustre: Skipped 14 previous similar messages [ 4356.612693] Lustre: server umount lustre-OST0000 complete [ 4367.077857] Lustre: server umount lustre-MDT0000 complete [ 4370.299944] LustreError: 146167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763336916 with bad export cookie 443064190086425083 [ 4370.307651] LustreError: 146167:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4370.307857] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.731265] Lustre: server umount lustre-MDT0001 complete [ 4386.348440] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4388.786497] Key type lgssc unregistered [ 4389.072785] LNet: 151687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.096323] LNetError: 151687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4389.127660] LNet: Removed LNI 192.168.204.129@tcp [ 4389.947155] Key type .llcrypt unregistered [ 4389.951335] Key type ._llcrypt unregistered [ 4410.292154] Key type ._llcrypt registered [ 4410.293660] Key type .llcrypt registered [ 4410.411313] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4411.150250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4411.181161] alg: No test for adler32 (adler32-zlib) [ 4412.236061] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4412.482752] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4414.151886] Key type lgssc registered [ 4415.420523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4423.817155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4423.844195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.419313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.516668] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4434.916277] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.925181] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4438.029134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4438.374832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4442.491962] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4447.965845] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.848274] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4459.695976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4459.958805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.997991] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4470.217749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4470.518512] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4476.887911] Lustre: Failing over lustre-MDT0000 [ 4477.063258] LustreError: 155310:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4477.135422] Lustre: server umount lustre-MDT0000 complete [ 4478.361591] LustreError: 153059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.459390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4479.463228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.476323] LustreError: 153775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.488789] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.602140] LustreError: 154272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.633668] LustreError: 154272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4489.219557] LustreError: 153042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.29@tcp arrived at 1763337035 with bad export cookie 10215807809902663585 [ 4489.241880] LustreError: 153042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4490.726506] LustreError: 154272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.765421] LustreError: 154272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4495.647098] Lustre: 152374:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763337026/real 1763337026] req@000000001bacde51 x1848992814096896/t0(0) o400->MGC192.168.204.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763337042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4495.665548] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.759890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.961271] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.987604] LustreError: 153063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4506.079855] LustreError: 152370:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000b33e950 x1848992814101504/t0(0) o250->MGC192.168.204.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4506.366578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.433930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4510.184930] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4511.718056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4511.735976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4512.123772] Lustre: lustre-MDT0000: Denying connection for new client 4ecb8748-8bcb-45a6-9c59-62cd4b5a4e46 (at 192.168.204.29@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4521.957651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.962750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4521.980670] Lustre: Skipped 2 previous similar messages [ 4522.005502] Lustre: Skipped 1 previous similar message [ 4527.074323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.088881] Lustre: Skipped 1 previous similar message [ 4532.194513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.205195] Lustre: Skipped 1 previous similar message [ 4534.239157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4534.285145] LustreError: 156246:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4534.294524] LustreError: 156246:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4534.428400] Lustre: server umount lustre-OST0000 complete [ 4537.649052] LustreError: 156448:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4537.661808] Lustre: 155813:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4537.669309] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4537.688414] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4537.694456] LustreError: 155813:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c9837379 x1848992814117376/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4537.728646] LustreError: 155813:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4537.750266] LustreError: 155813:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4537.768155] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 4537.981253] LustreError: 156448:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4537.984430] LustreError: 156448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4538.121867] Lustre: server umount lustre-MDT0000 complete [ 4541.707184] LustreError: 153044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337088 with bad export cookie 10215807809902663683 [ 4541.721259] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.722408] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4541.724477] LustreError: 153044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4541.733341] LustreError: 153064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4543.903822] LustreError: 156650:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4543.908065] LustreError: 156650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4544.108939] Lustre: server umount lustre-MDT0001 complete [ 4554.175946] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4556.665709] Key type lgssc unregistered [ 4557.007330] LNet: 157181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4557.013186] LNetError: 157181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4557.034709] LNet: Removed LNI 192.168.204.129@tcp [ 4557.764413] Key type .llcrypt unregistered [ 4557.766109] Key type ._llcrypt unregistered [ 4578.585215] Key type ._llcrypt registered [ 4578.587254] Key type .llcrypt registered [ 4578.658412] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4579.410306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4579.471887] alg: No test for adler32 (adler32-zlib) [ 4580.540517] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4580.774184] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4582.431616] Key type lgssc registered [ 4583.498557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4590.833427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4590.859889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.263850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.534317] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4601.828256] LustreError: 158568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4601.847377] LustreError: 158568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4602.546021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4602.894161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4606.199635] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4611.927310] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4615.291780] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4621.562966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4622.011963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.856600] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4632.706942] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4637.367107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4642.787755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4642.793989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4642.809245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4642.819621] Lustre: Skipped 1 previous similar message [ 4644.037183] LustreError: 160815:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4644.132713] Lustre: server umount lustre-OST0000 complete [ 4647.906326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.913451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4647.913796] Lustre: Skipped 1 previous similar message [ 4647.916011] Lustre: Skipped 1 previous similar message [ 4653.029946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4653.044537] Lustre: Skipped 2 previous similar messages [ 4653.565231] LustreError: 161016:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4653.602763] LustreError: 161016:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4653.710981] Lustre: server umount lustre-MDT0000 complete [ 4657.318182] LustreError: 158550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337203 with bad export cookie 3643071565411216772 [ 4657.321286] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4657.332939] LustreError: 158550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4657.504955] LustreError: 161217:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4657.511316] LustreError: 161217:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4657.670491] Lustre: server umount lustre-MDT0001 complete [ 4667.583755] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4670.307551] Key type lgssc unregistered [ 4670.709198] LNet: 161750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4670.720505] LNetError: 161750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4670.756699] LNet: Removed LNI 192.168.204.129@tcp [ 4671.543886] Key type .llcrypt unregistered [ 4671.545523] Key type ._llcrypt unregistered [ 4698.769714] Key type ._llcrypt registered [ 4698.775365] Key type .llcrypt registered [ 4698.889550] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4699.512518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4699.578269] alg: No test for adler32 (adler32-zlib) [ 4700.641855] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4700.856481] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4702.508263] Key type lgssc registered [ 4703.529408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4709.788042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4709.803317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4711.153777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4714.499984] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4720.616939] LustreError: 163411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.636698] LustreError: 163411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4721.604208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.012470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4725.080599] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4730.235752] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4733.605741] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4739.785847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4740.086074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.992158] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4750.235543] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4756.473669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4766.697755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4766.698090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4766.703350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4766.723483] Lustre: Skipped 1 previous similar message [ 4770.688101] LustreError: 165678:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4770.798078] Lustre: server umount lustre-OST0000 complete [ 4776.930502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.954717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4776.982329] Lustre: Skipped 1 previous similar message [ 4776.986903] Lustre: Skipped 2 previous similar messages [ 4780.759463] LustreError: 165881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4780.762205] LustreError: 165881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4780.993316] Lustre: server umount lustre-MDT0000 complete [ 4782.050533] LustreError: 163411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4782.058249] LustreError: 163411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4784.973863] LustreError: 163393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337331 with bad export cookie 10133899690073492942 [ 4784.987770] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4785.000562] LustreError: 163393:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4785.242366] LustreError: 166083:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4785.277542] LustreError: 166083:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4785.546988] Lustre: server umount lustre-MDT0001 complete [ 4796.144481] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4798.664327] Key type lgssc unregistered [ 4799.058330] LNet: 166615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4799.065927] LNetError: 166615:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4799.081782] LNet: Removed LNI 192.168.204.129@tcp [ 4799.804569] Key type .llcrypt unregistered [ 4799.807360] Key type ._llcrypt unregistered [ 4821.267077] Key type ._llcrypt registered [ 4821.268960] Key type .llcrypt registered [ 4821.359866] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4821.980766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4822.018099] alg: No test for adler32 (adler32-zlib) [ 4823.070135] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4823.272336] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4824.911270] Key type lgssc registered [ 4826.027380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4835.619822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4835.646750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4837.092280] Lustre: *** cfs_fail_loc=135, val=0*** [ 4837.193115] LustreError: 168119:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4837.202464] LustreError: 168119:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.129@tcp: cfg command failed: rc = -2 [ 4837.213533] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4837.213533] [ 4837.231242] LustreError: MGC192.168.204.129@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4837.242218] LustreError: 168080:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4837.246649] LustreError: 168080:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4837.250410] LustreError: 168080:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4837.261470] LustreError: 168089:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337383 with bad export cookie 4287878951908672137 [ 4837.267843] LustreError: 168080:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4837.411791] Lustre: server umount lustre-MDT0000 complete [ 4837.415858] LustreError: 168080:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4843.896494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.359755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.113920] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4853.732708] LustreError: 168493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.769187] LustreError: 168493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4857.017433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4857.325883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4860.647857] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4865.060660] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4868.337177] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.674786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4872.680730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.690418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4873.699713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4873.701844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.218286] LustreError: 169838:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4876.229803] LustreError: 169838:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4876.336974] Lustre: server umount lustre-MDT0000 complete [ 4878.822552] LustreError: 169558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.845556] LustreError: 169558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4879.233390] LustreError: 169211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337425 with bad export cookie 4287878951908672284 [ 4879.235054] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4879.239238] LustreError: 169211:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4879.352263] LustreError: 170039:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4879.355781] LustreError: 170039:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4879.451121] Lustre: server umount lustre-MDT0001 complete [ 4890.128491] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4893.288225] Key type lgssc unregistered [ 4893.598169] LNet: 170573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4893.610454] LNetError: 170573:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4893.630523] LNet: Removed LNI 192.168.204.129@tcp [ 4894.614986] Key type .llcrypt unregistered [ 4894.619927] Key type ._llcrypt unregistered [ 4914.202806] Key type ._llcrypt registered [ 4914.208692] Key type .llcrypt registered [ 4914.322386] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 4914.583199] Key type .llcrypt unregistered [ 4914.585414] Key type ._llcrypt unregistered [ 4929.120048] Key type ._llcrypt registered [ 4929.121253] Key type .llcrypt registered [ 4929.216233] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 4930.015232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4930.040615] alg: No test for adler32 (adler32-zlib) [ 4931.061570] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 4931.266741] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 4932.935941] Key type lgssc registered [ 4934.071370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4940.831776] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4940.859486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4956.447355] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4961.567230] LustreError: 172655:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.129@tcp: failed processing log, type 4: rc = -110 [ 4988.319204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4994.301221] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 4999.741401] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5006.103399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.548679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.625416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5010.104243] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5011.940462] LustreError: 172681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.956059] LustreError: 172681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5017.061132] LustreError: 172681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.713657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5017.974627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5021.145225] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5025.522101] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5028.800492] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5031.501987] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5042.661534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5042.674181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.682831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.681076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.690057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.729511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.549113] LustreError: 175332:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5047.679869] Lustre: server umount lustre-OST0000 complete [ 5053.922620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5053.935920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5053.963851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.966656] Lustre: Skipped 2 previous similar messages [ 5057.388282] LustreError: 175533:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5057.391291] LustreError: 175533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5057.569099] Lustre: server umount lustre-MDT0000 complete [ 5060.070369] LustreError: 173587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.084978] LustreError: 173587:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5060.925085] LustreError: 172663:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337607 with bad export cookie 14609373528833590210 [ 5060.947558] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5061.047644] LustreError: 175734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5061.060113] LustreError: 175734:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5061.377665] Lustre: server umount lustre-MDT0001 complete [ 5071.061379] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5073.924654] Key type lgssc unregistered [ 5074.173278] LNet: 176266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5074.183449] LNetError: 176266:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5074.202366] LNet: Removed LNI 192.168.204.129@tcp [ 5074.989178] Key type .llcrypt unregistered [ 5074.992233] Key type ._llcrypt unregistered [ 5097.680200] Key type ._llcrypt registered [ 5097.681799] Key type .llcrypt registered [ 5097.800990] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5098.767870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5098.823193] alg: No test for adler32 (adler32-zlib) [ 5099.834184] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5100.063132] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5101.751169] Key type lgssc registered [ 5102.845625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5109.813323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5109.828388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.243080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.433560] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5121.506820] LustreError: 177661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.516460] LustreError: 177661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5123.933816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5124.402056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.006584] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5133.715453] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5137.551861] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5144.123123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5144.641242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.170893] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5155.578628] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5155.902677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5161.267396] Lustre: Failing over lustre-MDT0000 [ 5161.467052] LustreError: 179971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5161.650827] Lustre: server umount lustre-MDT0000 complete [ 5163.413156] LustreError: 177655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5165.538112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5165.542723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.548443] LustreError: 178704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.051177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.059742] Lustre: Skipped 1 previous similar message [ 5168.495354] LustreError: 178705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5168.505838] LustreError: 178705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5173.627363] LustreError: 177656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5173.648600] LustreError: 177656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5180.026109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.182405] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5180.578716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.666050] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5183.855118] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5184.485139] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5186.026238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5186.068778] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5186.112677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5188.167549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5191.414516] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5204.960692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.965959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.979531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5205.384124] LustreError: 181173:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5205.396668] LustreError: 181173:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5205.522613] Lustre: server umount lustre-OST0000 complete [ 5211.616783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.617075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5211.637599] Lustre: Skipped 1 previous similar message [ 5211.640196] LustreError: Skipped 1 previous similar message [ 5211.648232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5211.652277] Lustre: Skipped 1 previous similar message [ 5216.737611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.838436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.847142] Lustre: Skipped 2 previous similar messages [ 5223.391178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5223.481888] LustreError: 181374:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5223.493209] LustreError: 181374:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5223.551512] Lustre: server umount lustre-MDT0000 complete [ 5225.953225] LustreError: 178704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.973095] LustreError: 178704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5227.016920] LustreError: 177640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337773 with bad export cookie 1560397672620631311 [ 5227.018462] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5227.027974] LustreError: 177640:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5227.175535] LustreError: 181577:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5227.187528] LustreError: 181577:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5227.379300] Lustre: server umount lustre-MDT0001 complete [ 5239.118353] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5241.689746] Key type lgssc unregistered [ 5241.956330] LNet: 182110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5241.964331] LNetError: 182110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5241.975557] LNet: Removed LNI 192.168.204.129@tcp [ 5242.753730] Key type .llcrypt unregistered [ 5242.755631] Key type ._llcrypt unregistered [ 5265.456106] Key type ._llcrypt registered [ 5265.457573] Key type .llcrypt registered [ 5265.568700] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5266.687729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5266.792906] alg: No test for adler32 (adler32-zlib) [ 5267.904842] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5268.089741] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5269.727188] Key type lgssc registered [ 5270.771778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5277.992166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5278.020300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.507460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.378441] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5288.942384] LustreError: 183504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.973392] LustreError: 183504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5291.545765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5291.886470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5295.636374] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5300.523849] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5304.276931] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5310.470358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5310.830077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5316.029049] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5321.267182] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5324.149733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5329.432979] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5330.598453] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5340.889809] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5349.859644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.865693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5349.878153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5349.893268] Lustre: Skipped 1 previous similar message [ 5354.419459] LustreError: 185914:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5354.550698] Lustre: server umount lustre-OST0000 complete [ 5358.562658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5358.569931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.581131] Lustre: Skipped 1 previous similar message [ 5358.589269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5360.097071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5360.102946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.755973] LustreError: 186116:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5363.759121] LustreError: 186116:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5363.851664] Lustre: server umount lustre-MDT0000 complete [ 5365.217866] LustreError: 183500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.234189] LustreError: 183500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5367.412508] LustreError: 185705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337913 with bad export cookie 482086027504514832 [ 5367.420167] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5367.426153] LustreError: 185705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5367.574683] LustreError: 186318:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5367.582739] LustreError: 186318:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5367.786780] Lustre: server umount lustre-MDT0001 complete [ 5377.474674] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5379.870732] Key type lgssc unregistered [ 5380.194865] LNet: 186850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5380.200882] LNetError: 186850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5381.229303] LNet: Removed LNI 192.168.204.129@tcp [ 5382.029712] Key type .llcrypt unregistered [ 5382.031562] Key type ._llcrypt unregistered [ 5404.245562] Key type ._llcrypt registered [ 5404.249229] Key type .llcrypt registered [ 5404.344275] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5405.265880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5405.474559] alg: No test for adler32 (adler32-zlib) [ 5406.578554] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5406.807184] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5408.505811] Key type lgssc registered [ 5409.601935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.951610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.972340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.464258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.065950] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5428.706428] LustreError: 188241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.714081] LustreError: 188241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5431.228619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5431.543987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5435.543825] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5441.244379] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5445.189455] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5451.819719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5452.194383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.480914] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5463.866570] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5464.496336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5473.051967] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5484.124752] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5495.412849] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5503.912944] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5515.745450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.746876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.760431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5515.769878] Lustre: Skipped 1 previous similar message [ 5519.302160] LustreError: 192077:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5519.444221] Lustre: server umount lustre-OST0000 complete [ 5523.938989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5523.943230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.951425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5523.954968] Lustre: Skipped 1 previous similar message [ 5525.984059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5525.990327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.784852] LustreError: 192279:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5528.787777] LustreError: 192279:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5529.097372] Lustre: server umount lustre-MDT0000 complete [ 5531.104122] LustreError: 188241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5531.138574] LustreError: 188241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5532.666581] LustreError: 188224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338079 with bad export cookie 13467621323733739392 [ 5532.670257] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.681697] LustreError: 188224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5532.888643] LustreError: 192480:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5532.907185] LustreError: 192480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5533.038285] Lustre: server umount lustre-MDT0001 complete [ 5543.608446] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5545.991089] Key type lgssc unregistered [ 5546.307662] LNet: 193012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5546.320159] LNetError: 193012:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5547.368578] LNet: Removed LNI 192.168.204.129@tcp [ 5548.163704] Key type .llcrypt unregistered [ 5548.166881] Key type ._llcrypt unregistered [ 5568.404432] Key type ._llcrypt registered [ 5568.405925] Key type .llcrypt registered [ 5568.492894] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5569.181905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5569.196637] alg: No test for adler32 (adler32-zlib) [ 5570.254593] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5570.464722] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5572.136128] Key type lgssc registered [ 5573.135278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.963880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.990744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.320981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.254839] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5591.220811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5591.657443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5595.870599] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5602.904403] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5607.271965] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5614.484376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5614.824641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5620.971147] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5626.009773] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5627.124436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5647.844592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.846810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.863550] LustreError: Skipped 1 previous similar message [ 5647.878976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5647.880807] Lustre: Skipped 1 previous similar message [ 5647.897893] Lustre: Skipped 1 previous similar message [ 5649.177320] LustreError: 196785:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5649.392567] Lustre: server umount lustre-OST0000 complete [ 5658.084771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.085774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5658.093951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5658.103383] Lustre: Skipped 1 previous similar message [ 5659.245481] LustreError: 196987:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5659.256545] LustreError: 196987:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5659.338502] Lustre: server umount lustre-MDT0000 complete [ 5662.783664] LustreError: 194366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338209 with bad export cookie 15467374486706141005 [ 5662.794179] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5662.810083] LustreError: 194366:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5663.001627] LustreError: 197188:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5663.009370] LustreError: 197188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5663.206797] Lustre: server umount lustre-MDT0001 complete [ 5672.976509] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5675.702967] Key type lgssc unregistered [ 5676.006275] LNet: 197720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5676.014795] LNetError: 197720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5677.033156] LNet: Removed LNI 192.168.204.129@tcp [ 5677.785509] Key type .llcrypt unregistered [ 5677.787338] Key type ._llcrypt unregistered [ 5696.684409] Key type ._llcrypt registered [ 5696.686142] Key type .llcrypt registered [ 5696.801378] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5698.031637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5698.099712] alg: No test for adler32 (adler32-zlib) [ 5699.188591] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5699.392945] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5701.039296] Key type lgssc registered [ 5702.123756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5709.172796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5709.191222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5710.704567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5713.729755] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5721.061782] LustreError: 198968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.077919] LustreError: 198968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5722.098777] LustreError: 198968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.471811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5722.869799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5726.736692] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5731.900867] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5736.182833] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5743.075806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5743.380787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.452612] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5753.399246] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5756.724368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5772.256362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5772.259704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.273922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.953775] LustreError: 201367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5776.054426] Lustre: server umount lustre-OST0000 complete [ 5779.424832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5779.432920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.451583] Lustre: Skipped 1 previous similar message [ 5779.454399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.457218] Lustre: Skipped 1 previous similar message [ 5782.496866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5782.499460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.520308] LustreError: 201568:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5785.528137] LustreError: 201568:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5785.642250] Lustre: server umount lustre-MDT0000 complete [ 5787.617927] LustreError: 198963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5787.641024] LustreError: 198963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5789.313774] LustreError: 198949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338335 with bad export cookie 9019589637008060015 [ 5789.319946] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.325257] LustreError: 198949:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5789.532841] LustreError: 201771:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5789.543918] LustreError: 201771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5789.936757] Lustre: server umount lustre-MDT0001 complete [ 5800.521825] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 5803.426033] Key type lgssc unregistered [ 5803.728784] LNet: 202303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.740249] LNetError: 202303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5803.752181] LNet: Removed LNI 192.168.204.129@tcp [ 5804.627460] Key type .llcrypt unregistered [ 5804.628902] Key type ._llcrypt unregistered [ 5823.140142] Key type ._llcrypt registered [ 5823.143725] Key type .llcrypt registered [ 5823.253904] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 5824.000106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5824.023781] alg: No test for adler32 (adler32-zlib) [ 5825.130683] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 5825.353946] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 5827.031856] Key type lgssc registered [ 5828.412366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5836.619048] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5836.644147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.268211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.518603] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5848.551145] LustreError: 203568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.593764] LustreError: 203568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5851.076628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5851.515839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5855.558861] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5861.800599] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5865.263261] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5871.541632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5871.918649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.339832] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5881.697617] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5884.208523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5895.111228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5895.156075] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5895.362508] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5901.234069] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 5906.929691] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5908.692295] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5920.104946] Lustre: Permanently deactivating lustre-OST0001 [ 5920.127182] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5926.473235] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5934.921994] Lustre: Permanently reactivating lustre-OST0001 [ 5934.962248] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5934.965566] Lustre: Skipped 2 previous similar messages [ 5943.939951] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5943.950386] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5943.957993] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5943.972978] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5944.005658] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5948.158166] LustreError: 207002:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5948.241839] Lustre: server umount lustre-OST0001 complete [ 5949.422117] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5949.425963] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.431785] Lustre: Skipped 1 previous similar message [ 5949.434378] LustreError: 206503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.996642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5953.000654] LustreError: Skipped 1 previous similar message [ 5953.007943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.032692] Lustre: Skipped 1 previous similar message [ 5953.052870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.535047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.538655] LustreError: 206637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.582257] LustreError: 206637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5959.652185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.656566] LustreError: 205147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5959.656655] Lustre: Skipped 2 previous similar messages [ 5959.688280] LustreError: 205147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5964.773885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.786073] LustreError: 205148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5964.800624] LustreError: 205148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5967.327205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5967.384029] LustreError: 207203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5967.388814] LustreError: 207203:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5967.533689] Lustre: server umount lustre-OST0000 complete [ 5974.499668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5974.504929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5974.515976] Lustre: Skipped 1 previous similar message [ 5974.520937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5974.524014] Lustre: Skipped 1 previous similar message [ 5985.251305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.254711] Lustre: Skipped 5 previous similar messages [ 5986.271199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5986.353560] LustreError: 207405:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5986.365145] LustreError: 207405:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5986.472886] Lustre: server umount lustre-MDT0000 complete [ 5989.837284] LustreError: 203531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338536 with bad export cookie 2672747115025092106 [ 5989.838438] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5989.844540] LustreError: 203531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5990.012968] LustreError: 207608:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5990.021576] LustreError: 207608:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5990.202182] Lustre: server umount lustre-MDT0001 complete [ 6000.339557] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6003.114448] Key type lgssc unregistered [ 6003.448707] LNet: 208140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6003.460058] LNetError: 208140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6003.486136] LNet: Removed LNI 192.168.204.129@tcp [ 6004.299590] Key type .llcrypt unregistered [ 6004.307320] Key type ._llcrypt unregistered [ 6033.921416] Key type ._llcrypt registered [ 6033.922906] Key type .llcrypt registered [ 6034.029613] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 6050.214929] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6051.288147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6051.324509] alg: No test for adler32 (adler32-zlib) [ 6052.476514] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6052.808847] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6054.519334] Key type lgssc registered [ 6055.668565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6065.416145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6072.402435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6078.931975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6084.934704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.743923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6091.796272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6091.823615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.044137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6093.076180] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6093.206024] Lustre: lustre-MDT0000: new disk, initializing [ 6093.294341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.308740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6097.531366] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6107.118237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6107.194328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6107.270959] Lustre: 212088:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6107.320046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6107.323092] Lustre: Skipped 1 previous similar message [ 6107.389969] Lustre: lustre-MDT0001: new disk, initializing [ 6107.512682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6107.544428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6107.559802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6110.693775] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6117.422188] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6121.165758] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6128.208304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6128.287382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6128.706779] Lustre: lustre-OST0000: new disk, initializing [ 6128.710997] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6128.847435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.037961] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6136.359061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6136.368966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6136.481061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6140.779467] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6144.127497] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6144.340093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6147.595945] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6147.793791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6152.160905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6152.165203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.168810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6152.171503] Lustre: Skipped 1 previous similar message [ 6155.581150] LustreError: 215026:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6155.705918] Lustre: server umount lustre-OST0000 complete [ 6162.400651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.410091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.414761] Lustre: Skipped 1 previous similar message [ 6162.421821] Lustre: Skipped 1 previous similar message [ 6165.546064] LustreError: 215227:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6165.558132] LustreError: 215227:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6165.776807] Lustre: server umount lustre-MDT0000 complete [ 6169.448048] LustreError: 212868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338715 with bad export cookie 10348021796399814285 [ 6169.455542] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6169.507967] LustreError: 212868:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6169.717826] LustreError: 215429:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6169.778086] LustreError: 215429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6170.123611] Lustre: server umount lustre-MDT0001 complete [ 6179.678305] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 19:18:44 (1763338724) [ 6186.976468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.304772] LustreError: 216102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.324994] LustreError: 216102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6187.367881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6191.335734] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6192.608481] LustreError: 216102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6196.716719] LustreError: 216101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6199.958881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6200.396626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.558263] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6210.532406] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6215.028824] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6223.230769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6223.771419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6229.602254] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6235.218099] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6238.573085] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6243.043373] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6248.738120] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6255.286238] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6273.060155] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6273.068845] Lustre: Skipped 1 previous similar message [ 6293.363212] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6293.370777] Lustre: Skipped 2 previous similar messages [ 6332.932540] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6332.936305] Lustre: Skipped 4 previous similar messages [ 6400.705659] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6400.709282] Lustre: Skipped 7 previous similar messages [ 6410.212365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6410.226221] LustreError: Skipped 1 previous similar message [ 6410.229427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.241789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6410.244724] Lustre: Skipped 1 previous similar message [ 6411.232329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.546231] LustreError: 219587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6412.568641] LustreError: 219587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6412.731922] Lustre: server umount lustre-OST0000 complete [ 6420.451761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6420.457995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.468196] Lustre: Skipped 1 previous similar message [ 6420.484410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6420.487551] Lustre: Skipped 2 previous similar messages [ 6422.758484] LustreError: 219788:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6422.763118] LustreError: 219788:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6422.884811] Lustre: server umount lustre-MDT0000 complete [ 6426.595129] LustreError: 216102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.636421] LustreError: 216102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6427.042380] LustreError: 218302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338973 with bad export cookie 10348021796399815167 [ 6427.043615] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.061610] LustreError: 218302:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6427.366285] Lustre: server umount lustre-MDT0001 complete [ 6436.341629] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6439.128802] Key type lgssc unregistered [ 6439.498804] LNet: 220523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6439.507260] LNetError: 220523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6440.556474] LNet: Removed LNI 192.168.204.129@tcp [ 6441.385933] Key type .llcrypt unregistered [ 6441.387441] Key type ._llcrypt unregistered [ 6461.547539] Key type ._llcrypt registered [ 6461.549479] Key type .llcrypt registered [ 6461.622783] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6462.431237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6462.521621] alg: No test for adler32 (adler32-zlib) [ 6463.539680] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6463.753616] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6465.415222] Key type lgssc registered [ 6466.298796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.820346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6472.832892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6474.173871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6477.687990] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6484.450997] LustreError: 221930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.461698] LustreError: 221930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6485.846453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6486.187737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6489.594267] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6494.363253] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6497.281532] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6502.819632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6503.177324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.509201] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_default_debug -1 all [ 6513.586334] Lustre: DEBUG MARKER: oleg429-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6535.141165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.143988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.145312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.155807] Lustre: Skipped 1 previous similar message [ 6535.163153] Lustre: Skipped 1 previous similar message [ 6540.259575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6540.263634] Lustre: Skipped 1 previous similar message [ 6541.052161] LustreError: 224338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6541.145899] Lustre: server umount lustre-OST0000 complete [ 6545.380586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6545.385306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6545.392837] Lustre: Skipped 1 previous similar message [ 6545.398031] Lustre: Skipped 1 previous similar message [ 6549.710195] LustreError: 224539:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6549.713984] LustreError: 224539:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6549.827887] Lustre: server umount lustre-MDT0000 complete [ 6550.497282] LustreError: 221931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6550.507557] LustreError: 221931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6553.045970] LustreError: 221911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339099 with bad export cookie 15013819286037022169 [ 6553.051847] LustreError: MGC192.168.204.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.053205] LustreError: 221911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6553.172189] LustreError: 224740:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6553.182609] LustreError: 224740:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6553.439260] Lustre: server umount lustre-MDT0001 complete [ 6561.034889] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6563.346960] Key type lgssc unregistered [ 6563.585107] LNet: 225272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6563.591972] LNetError: 225272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6563.610083] LNet: Removed LNI 192.168.204.129@tcp [ 6564.293240] Key type .llcrypt unregistered [ 6564.295672] Key type ._llcrypt unregistered [ 6581.548068] Key type ._llcrypt registered [ 6581.549502] Key type .llcrypt registered [ 6581.615165] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6581.835402] Key type .llcrypt unregistered [ 6581.837098] Key type ._llcrypt unregistered [ 6600.341656] Key type ._llcrypt registered [ 6600.343608] Key type .llcrypt registered [ 6600.432518] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing set_hostid [ 6612.906079] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing load_modules_local [ 6613.505758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6613.567556] alg: No test for adler32 (adler32-zlib) [ 6614.593752] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 6614.776689] LNet: Added LNI 192.168.204.129@tcp [8/256/0/180] [ 6616.431191] Key type lgssc registered [ 6617.279043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.241867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6632.077256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6638.640576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.257180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.262544] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6365 sec ========== 19:26:33 (1763339193) [ 6650.332375] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:26:35 (1763339195) === [ 6654.379977] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:26:39 (1763339199) === [ 6692.211568] Lustre: DEBUG MARKER: oleg429-server.virtnet: executing unload_modules_local [ 6694.653269] Key type lgssc unregistered [ 6694.917654] LNet: 231380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6694.934865] LNetError: 231380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6694.955909] LNet: Removed LNI 192.168.204.129@tcp [ 6695.688207] Key type .llcrypt unregistered [ 6695.689722] Key type ._llcrypt unregistered