[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507099605 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003013] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007014] pid_max: default: 32768 minimum: 301 [ 0.008134] LSM: Security Framework initializing [ 0.009048] Yama: becoming mindful. [ 0.010030] SELinux: Initializing. [ 0.011069] *** VALIDATE selinux *** [ 0.019687] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023767] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027106] *** VALIDATE tmpfs *** [ 0.029058] *** VALIDATE proc *** [ 0.030234] *** VALIDATE cgroup *** [ 0.031023] *** VALIDATE cgroup2 *** [ 0.032271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035031] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039309] debug: unmapping init [mem 0xffffffffb4e59000-0xffffffffb4e60fff] [ 0.041169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042751] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045014] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.049009] ... event mask: 000000070000000f [ 0.051139] rcu: Hierarchical SRCU implementation. [ 0.053337] smp: Bringing up secondary CPUs ... [ 0.054646] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.060293] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151024] node 0 deferred pages initialised in 86ms [ 0.154131] devtmpfs: initialized [ 0.155198] x86/mm: Memory block size: 128MB [ 0.157405] gcov: version magic: 0x41383552 [ 0.159116] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161232] pinctrl core: initialized pinctrl subsystem [ 0.162201] [ 0.162772] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164011] ** ** [ 0.165010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166013] ** ** [ 0.167014] ** This means that this kernel is built to expose internal ** [ 0.168014] ** IOMMU data structures, which may compromise security on ** [ 0.169014] ** your system. ** [ 0.170013] ** ** [ 0.171015] ** If you see this message and you are not debugging the ** [ 0.172012] ** kernel, report this immediately to your vendor! ** [ 0.173012] ** ** [ 0.174012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175015] ************************************************************* [ 0.176639] NET: Registered protocol family 16 [ 0.177440] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180446] cpuidle: using governor menu [ 0.183111] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186879] PCI: Using configuration type 1 for base access [ 0.189164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201086] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204203] cryptd: max_cpu_qlen set to 1000 [ 0.206259] ACPI: Added _OSI(Module Device) [ 0.207032] ACPI: Added _OSI(Processor Device) [ 0.208000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218313] ACPI: Interpreter enabled [ 0.220087] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221015] ACPI: Using IOAPIC for interrupt routing [ 0.223180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.226455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236297] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243112] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248315] acpiphp: Slot [2] registered [ 0.249237] acpiphp: Slot [3] registered [ 0.251128] acpiphp: Slot [4] registered [ 0.253138] acpiphp: Slot [5] registered [ 0.254342] acpiphp: Slot [6] registered [ 0.256202] acpiphp: Slot [7] registered [ 0.258157] acpiphp: Slot [8] registered [ 0.259182] acpiphp: Slot [9] registered [ 0.261208] acpiphp: Slot [10] registered [ 0.263170] acpiphp: Slot [11] registered [ 0.264463] acpiphp: Slot [12] registered [ 0.266132] acpiphp: Slot [13] registered [ 0.268132] acpiphp: Slot [14] registered [ 0.269133] acpiphp: Slot [15] registered [ 0.271160] acpiphp: Slot [16] registered [ 0.273115] acpiphp: Slot [17] registered [ 0.274170] acpiphp: Slot [18] registered [ 0.276129] acpiphp: Slot [19] registered [ 0.277132] acpiphp: Slot [20] registered [ 0.279124] acpiphp: Slot [21] registered [ 0.280129] acpiphp: Slot [22] registered [ 0.282162] acpiphp: Slot [23] registered [ 0.283110] acpiphp: Slot [24] registered [ 0.285119] acpiphp: Slot [25] registered [ 0.286109] acpiphp: Slot [26] registered [ 0.288140] acpiphp: Slot [27] registered [ 0.289102] acpiphp: Slot [28] registered [ 0.291131] acpiphp: Slot [29] registered [ 0.292106] acpiphp: Slot [30] registered [ 0.294104] acpiphp: Slot [31] registered [ 0.295076] PCI host bridge to bus 0000:00 [ 0.296025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306033] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.308031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314691] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.319412] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328769] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334539] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345591] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.347845] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354534] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372024] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.379883] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384781] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.392027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417020] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.426863] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.433016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.459019] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.471769] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.489067] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508026] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.517298] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.527021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562025] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.573639] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.612023] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.625351] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.634018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.641030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.668989] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.671438] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.673466] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.676432] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679308] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683166] iommu: Default domain type: Passthrough [ 0.685777] SCSI subsystem initialized [ 0.687155] ACPI: bus type USB registered [ 0.689118] usbcore: registered new interface driver usbfs [ 0.691090] usbcore: registered new interface driver hub [ 0.693124] usbcore: registered new device driver usb [ 0.695206] pps_core: LinuxPPS API ver. 1 registered [ 0.697012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.700092] PTP clock support registered [ 0.703043] EDAC MC: Ver: 3.0.0 [ 0.704168] PCI: Using ACPI for IRQ routing [ 0.706887] NetLabel: Initializing [ 0.708012] NetLabel: domain hash size = 128 [ 0.709010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711083] NetLabel: unlabeled traffic allowed by default [ 0.714061] vgaarb: loaded [ 0.715279] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.716010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721369] clocksource: Switched to clocksource kvm-clock [ 0.829189] VFS: Disk quotas dquot_6.6.0 [ 0.830652] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.832413] *** VALIDATE ramfs *** [ 0.833250] *** VALIDATE hugetlbfs *** [ 0.834303] pnp: PnP ACPI init [ 0.836469] pnp: PnP ACPI: found 6 devices [ 0.855916] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858977] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861194] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.863328] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.864980] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.867609] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.870563] NET: Registered protocol family 2 [ 0.873051] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.877943] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.881890] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.886475] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.890037] TCP: Hash tables configured (established 65536 bind 65536) [ 0.892280] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.895369] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898435] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901271] NET: Registered protocol family 1 [ 0.903858] RPC: Registered named UNIX socket transport module. [ 0.906053] RPC: Registered udp transport module. [ 0.907690] RPC: Registered tcp transport module. [ 0.909378] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911907] NET: Registered protocol family 44 [ 0.913538] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.915652] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.917620] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.919672] PCI: CLS 0 bytes, default 64 [ 0.921189] Unpacking initramfs... [ 2.368207] debug: unmapping init [mem 0xffff9e60bcc54000-0xffff9e60bffbffff] [ 2.372859] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.374981] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.377984] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.875872] Initialise system trusted keyrings [ 2.877840] Key type blacklist registered [ 2.881334] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.892397] zbud: loaded [ 2.895862] *** VALIDATE nfs *** [ 2.897422] *** VALIDATE nfs4 *** [ 2.899741] pstore: using deflate compression [ 2.903153] Platform Keyring initialized [ 3.043479] NET: Registered protocol family 38 [ 3.045597] Key type asymmetric registered [ 3.046958] Asymmetric key parser 'x509' registered [ 3.049150] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.052320] io scheduler mq-deadline registered [ 3.054277] io scheduler kyber registered [ 3.056099] io scheduler bfq registered [ 3.057557] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.061065] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.063327] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.066529] ACPI: Power Button [PWRF] [ 3.169833] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.262700] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.443845] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.535274] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.723794] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.756896] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.793044] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.799348] Non-volatile memory driver v1.3 [ 3.800835] Linux agpgart interface v0.103 [ 3.833938] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.837761] vda: detected capacity change from 0 to 68530176 [ 3.853930] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.856767] vdb: detected capacity change from 0 to 1073741824 [ 3.875446] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.879073] vdc: detected capacity change from 0 to 2621440000 [ 3.893172] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.895901] vdd: detected capacity change from 0 to 2621440000 [ 3.915577] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.918383] vde: detected capacity change from 0 to 4294967296 [ 3.931710] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.934062] vdf: detected capacity change from 0 to 4294967296 [ 3.941271] libphy: Fixed MDIO Bus: probed [ 3.949812] usbcore: registered new interface driver usbserial_generic [ 3.952153] usbserial: USB Serial support registered for generic [ 3.954650] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.959440] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.961246] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.963900] mousedev: PS/2 mouse device common for all mice [ 3.969228] rtc_cmos 00:05: RTC can wake from S4 [ 3.971434] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.973204] rtc_cmos 00:05: registered as rtc0 [ 3.980591] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.982987] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.983714] intel_pstate: CPU model not supported [ 3.987126] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.991716] hid: raw HID events driver (C) Jiri Kosina [ 3.993820] usbcore: registered new interface driver usbhid [ 3.995677] usbhid: USB HID core driver [ 3.997244] drop_monitor: Initializing network drop monitor service [ 3.999518] Initializing XFRM netlink socket [ 4.001692] NET: Registered protocol family 10 [ 4.004856] Segment Routing with IPv6 [ 4.006637] NET: Registered protocol family 17 [ 4.008944] mpls_gso: MPLS GSO support [ 4.013554] RAS: Correctable Errors collector initialized. [ 4.015883] AVX version of gcm_enc/dec engaged. [ 4.017572] AES CTR mode by8 optimization enabled [ 4.107111] sched_clock: Marking stable (4107077877, 0)->(4997142899, -890065022) [ 4.110214] registered taskstats version 1 [ 4.113082] Loading compiled-in X.509 certificates [ 4.115525] zswap: loaded using pool lzo/zbud [ 4.138496] Key type big_key registered [ 4.151499] Key type encrypted registered [ 4.152809] ima: No TPM chip found, activating TPM-bypass! [ 4.154630] ima: Allocated hash algorithm: sha1 [ 4.156433] ima: No architecture policies found [ 4.157825] evm: Initialising EVM extended attributes: [ 4.159506] evm: security.selinux [ 4.160415] evm: security.ima [ 4.161240] evm: security.capability [ 4.162636] evm: HMAC attrs: 0x1 [ 4.165670] rtc_cmos 00:05: setting system clock to 2025-11-17 01:48:40 UTC (1763344120) [ 4.171283] debug: unmapping init [mem 0xffffffffb5e03000-0xffffffffb5ffffff] [ 4.173776] debug: unmapping init [mem 0xffffffffb4b82000-0xffffffffb4e58fff] [ 4.183118] Write protecting the kernel read-only data: 28672k [ 4.187026] debug: unmapping init [mem 0xffffffffb3203000-0xffffffffb33fffff] [ 4.190068] debug: unmapping init [mem 0xffffffffb3b14000-0xffffffffb3bfffff] [ 4.233429] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.240273] systemd[1]: Detected virtualization kvm. [ 4.241762] systemd[1]: Detected architecture x86-64. [ 4.243237] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.267936] systemd[1]: No hostname configured. [ 4.269681] systemd[1]: Set hostname to . [ 4.271361] random: systemd: uninitialized urandom read (16 bytes read) [ 4.273260] systemd[1]: Initializing machine ID from random generator. [ 4.317674] random: ln: uninitialized urandom read (6 bytes read) [ 4.402070] random: systemd: uninitialized urandom read (16 bytes read) [ 4.404364] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.409616] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.415020] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [[ 4.552186] random: fast init done  OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.074074] device-mapper: uevent: version 1.0.3 [ 5.076502] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.951851] virtio_net virtio0 ens2: renamed from eth0 [ 6.031176] scsi host0: ata_piix [ 6.089387] scsi host1: ata_piix [ 6.090751] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.092636] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.874930] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.675982] random: crng init done [ 10.678357] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.243513] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.469919] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.754504] SELinux: Disabled at runtime. [ 12.817259] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.825087] systemd[1]: Detected virtualization kvm. [ 12.826934] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.367887] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.371596] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.376905] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.383524] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.386969] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.393080] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.396769] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ 13.492609] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.934982] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.285734] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.287168] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.471515] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.482917] EDAC sbridge: Ver: 1.1.2 [ 16.242725] Key type dns_resolver registered [ 16.575788] NFS: Registering the id_resolver key type [ 16.578202] Key type id_resolver registered [ 16.580027] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg443-server login: [ 44.172379] libcfs: loading out-of-tree module taints kernel. [ 44.185377] Key type ._llcrypt registered [ 44.186856] Key type .llcrypt registered [ 44.241654] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 53.505906] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 54.390528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.408694] alg: No test for adler32 (adler32-zlib) [ 55.615459] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 56.156745] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 57.848220] Key type lgssc registered [ 59.088323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.006932] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.946402] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.026226] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.617308] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.256655] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.839952] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 117.401443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.450361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 117.470063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.697931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 118.726752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 118.791718] Lustre: lustre-MDT0000: new disk, initializing [ 118.881836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.915292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.950558] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.650602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.740983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 133.797669] Lustre: 6486:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 133.823616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 133.829401] Lustre: Skipped 1 previous similar message [ 133.968787] Lustre: lustre-MDT0001: new disk, initializing [ 134.045918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 134.110600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 134.127367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 138.907766] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.623314] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 152.607608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.679765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 152.922135] Lustre: lustre-OST0000: new disk, initializing [ 152.932918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 153.026601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.056326] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.811553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 160.825667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 160.878342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 171.723943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.822706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.885971] Lustre: lustre-OST0001: new disk, initializing [ 171.894858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 171.952157] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.366676] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.192572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 178.196623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 178.238299] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 185.840512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 191.656088] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 201.698388] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing check_logdir /tmp/testlogs/ [ 205.126901] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing yml_node [ 210.201883] Lustre: DEBUG MARKER: Client: 2.16.61.46 [ 212.866963] Lustre: DEBUG MARKER: MDS: 2.16.61.46 [ 213.716358] hrtimer: interrupt took 3846373 ns [ 215.299848] Lustre: DEBUG MARKER: OSS: 2.16.61.46 [ 216.768450] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 20:52:12 EST 2025 [ 231.376356] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 232.650784] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 249.828431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 249.830202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.844611] Lustre: Skipped 3 previous similar messages [ 249.876755] Lustre: Skipped 3 previous similar messages [ 253.719280] LustreError: 12187:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 253.835754] Lustre: server umount lustre-MDT0000 complete [ 256.369029] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344372 with bad export cookie 15458495642741096965 [ 256.373802] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 256.378258] LustreError: 6476:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 260.070985] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.071129] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.074694] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 260.078460] LustreError: 6492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 260.087531] Lustre: Skipped 1 previous similar message [ 261.088744] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 261.088821] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 261.101619] Lustre: Skipped 2 previous similar messages [ 262.570345] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 262.573252] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 262.709476] Lustre: server umount lustre-MDT0001 complete [ 265.468585] LustreError: 12591:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 265.474919] LustreError: 12591:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 265.509570] Lustre: server umount lustre-OST0000 complete [ 268.029354] LustreError: 12791:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 268.031626] LustreError: 12791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 268.121751] Lustre: server umount lustre-OST0001 complete [ 271.985404] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 278.982453] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 286.632976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 291.459717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 296.485820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 302.019344] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.498877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.560123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 308.840091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 308.860273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 308.922775] Lustre: lustre-MDT0000: new disk, initializing [ 308.979718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.002957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 312.412718] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 321.960136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.042407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 322.176153] Lustre: 14974:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 322.241803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 322.245685] Lustre: Skipped 1 previous similar message [ 322.414540] Lustre: lustre-MDT0001: new disk, initializing [ 322.537506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 322.598058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 322.608154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 326.625129] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 334.405525] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 337.437880] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 343.569153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.631789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 343.878091] Lustre: lustre-OST0000: new disk, initializing [ 343.885934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 343.982195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.637362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 345.663867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 345.821886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 349.341642] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 356.334438] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 360.218321] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 360.428368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 363.731961] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 363.872227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 366.563324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 366.563628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.571447] LustreError: Skipped 1 previous similar message [ 366.594106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 366.599565] Lustre: Skipped 1 previous similar message [ 371.520969] LustreError: 17916:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 371.534552] LustreError: 17916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 371.647815] Lustre: server umount lustre-OST0000 complete [ 376.805770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.807505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 376.813781] Lustre: Skipped 2 previous similar messages [ 376.818388] Lustre: Skipped 1 previous similar message [ 387.045843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.059499] Lustre: Skipped 3 previous similar messages [ 389.089564] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 389.216469] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 389.232201] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 389.347075] Lustre: server umount lustre-MDT0000 complete [ 392.161528] LustreError: 14980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.172845] LustreError: 14980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 392.374979] LustreError: 14965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344508 with bad export cookie 15458495642741098498 [ 392.382957] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.387695] LustreError: 14965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 392.670956] Lustre: server umount lustre-MDT0001 complete [ 395.240676] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:55:10 (1763344510) [ 399.772409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.112304] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.185600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.172182] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 405.473521] LustreError: 18849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.479786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 414.498469] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 419.324311] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.313151] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 427.125177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 431.541461] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 436.364514] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 455.149528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.153684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.155317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 455.180643] Lustre: Skipped 1 previous similar message [ 459.783222] LustreError: 21117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 459.794095] LustreError: 21117:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 459.889922] Lustre: server umount lustre-OST0000 complete [ 465.379854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 469.284339] Lustre: server umount lustre-MDT0000 complete [ 470.498461] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.516093] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 472.192537] LustreError: 18828:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344588 with bad export cookie 15458495642741099373 [ 472.193894] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 472.201344] LustreError: 18828:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 472.490654] Lustre: server umount lustre-MDT0001 complete [ 479.258500] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 480.864038] Key type lgssc unregistered [ 481.053784] LNet: 22052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.059032] LNetError: 22052:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.066514] LNet: Removed LNI 192.168.204.143@tcp [ 481.620162] Key type .llcrypt unregistered [ 481.621937] Key type ._llcrypt unregistered [ 499.451193] Key type ._llcrypt registered [ 499.452603] Key type .llcrypt registered [ 499.567818] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 500.902126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 500.946113] alg: No test for adler32 (adler32-zlib) [ 501.919534] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 502.184296] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 503.800599] Key type lgssc registered [ 504.879635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 511.396606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 511.410846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.813081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.226651] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 523.031183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 523.292065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 526.793575] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 532.318954] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 535.644554] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 541.317802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 541.575363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.371293] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 550.888683] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 557.951572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 568.293381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.299986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.300552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.312370] Lustre: Skipped 1 previous similar message [ 570.748158] LustreError: 26111:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 570.839384] Lustre: server umount lustre-OST0000 complete [ 574.435893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.439464] LustreError: Skipped 1 previous similar message [ 574.441654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.446960] Lustre: Skipped 1 previous similar message [ 574.449666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.532314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 578.537990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.300995] LustreError: 26314:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 580.321070] LustreError: 26314:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 580.416853] Lustre: server umount lustre-MDT0000 complete [ 583.650270] LustreError: 23422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.689454] LustreError: 23422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 584.145501] LustreError: 24141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344700 with bad export cookie 8910446175495894394 [ 584.148266] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 584.182487] LustreError: 24141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 584.295729] LustreError: 26515:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 584.306562] LustreError: 26515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 584.526874] Lustre: server umount lustre-MDT0001 complete [ 593.949877] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 596.499200] Key type lgssc unregistered [ 596.727621] LNet: 27046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.731401] LNetError: 27046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.799331] LNet: Removed LNI 192.168.204.143@tcp [ 598.506076] Key type .llcrypt unregistered [ 598.512448] Key type ._llcrypt unregistered [ 620.979967] Key type ._llcrypt registered [ 620.988328] Key type .llcrypt registered [ 621.139588] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 622.243697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.276061] alg: No test for adler32 (adler32-zlib) [ 623.409667] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 623.664360] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 625.400833] Key type lgssc registered [ 626.527183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.671291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 633.692690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.206387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.881820] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 645.602961] LustreError: 28439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 645.615371] LustreError: 28439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 645.660182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 645.940984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 649.016089] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 652.533883] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 655.090597] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 666.025789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 666.349539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.238524] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 676.406845] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 680.687494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 686.050229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 686.052049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.054762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 686.059783] Lustre: Skipped 1 previous similar message [ 686.064920] Lustre: Skipped 1 previous similar message [ 687.189248] LustreError: 31031:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 687.352113] Lustre: server umount lustre-OST0000 complete [ 691.174160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.179787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 691.190307] Lustre: Skipped 1 previous similar message [ 691.202320] Lustre: Skipped 1 previous similar message [ 696.290135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.308839] Lustre: Skipped 1 previous similar message [ 696.960215] LustreError: 31232:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 696.967348] LustreError: 31232:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 697.323648] Lustre: server umount lustre-MDT0000 complete [ 700.755886] LustreError: 28421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344817 with bad export cookie 6654732433192951570 [ 700.758845] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.768025] LustreError: 28421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 700.890785] LustreError: 31433:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 700.894072] LustreError: 31433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 701.066158] Lustre: server umount lustre-MDT0001 complete [ 712.199484] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 715.055661] Key type lgssc unregistered [ 715.328876] LNet: 31966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.339751] LNetError: 31966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.370768] LNet: Removed LNI 192.168.204.143@tcp [ 716.123510] Key type .llcrypt unregistered [ 716.128384] Key type ._llcrypt unregistered [ 738.788690] Key type ._llcrypt registered [ 738.790426] Key type .llcrypt registered [ 738.879329] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 739.501320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 739.556126] alg: No test for adler32 (adler32-zlib) [ 740.572530] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 740.767521] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 742.482798] Key type lgssc registered [ 743.268448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.288833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 751.320842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 752.806903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.668960] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 763.368922] LustreError: 33373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.398393] LustreError: 33373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 766.044656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 766.372361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 770.053180] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 775.232337] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 778.603715] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 785.105533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 785.460191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.289734] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 795.045652] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 797.746083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 807.907044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.912735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 807.913043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 807.936987] Lustre: Skipped 1 previous similar message [ 812.212639] LustreError: 35624:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 812.330894] Lustre: server umount lustre-OST0000 complete [ 817.638505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 817.642227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.652296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 817.655219] Lustre: Skipped 1 previous similar message [ 821.545129] LustreError: 35825:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 821.547615] LustreError: 35825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 821.611308] Lustre: server umount lustre-MDT0000 complete [ 823.281314] LustreError: 33375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.302883] LustreError: 33375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 824.798397] LustreError: 33355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344941 with bad export cookie 9848460179051352373 [ 824.807982] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 824.920235] LustreError: 36027:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 824.927021] LustreError: 36027:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 825.102377] Lustre: server umount lustre-MDT0001 complete [ 833.219470] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 835.230264] Key type lgssc unregistered [ 835.433971] LNet: 36558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.439305] LNetError: 36558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.452903] LNet: Removed LNI 192.168.204.143@tcp [ 835.967095] Key type .llcrypt unregistered [ 835.968773] Key type ._llcrypt unregistered [ 854.496738] Key type ._llcrypt registered [ 854.499099] Key type .llcrypt registered [ 854.602428] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 855.200303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.301202] alg: No test for adler32 (adler32-zlib) [ 856.303271] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 856.478951] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 858.168698] Key type lgssc registered [ 859.132996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 865.603442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 865.618940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.023132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.983916] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 875.947645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 876.463636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 879.943335] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 886.667954] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 891.026822] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 897.898132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 898.238398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.182902] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 908.595785] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 912.565689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 919.710371] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 922.081195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.084326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.111310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.109177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.118045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.125319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.129288] Lustre: Skipped 1 previous similar message [ 928.228030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.230872] Lustre: Skipped 2 previous similar messages [ 932.320174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 932.363156] LustreError: 40177:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 932.502878] Lustre: server umount lustre-OST0000 complete [ 953.317180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.338123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.363152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.379612] Lustre: Skipped 1 previous similar message [ 955.565533] LustreError: 40555:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 955.570091] LustreError: 40555:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 955.643080] Lustre: server umount lustre-MDT0000 complete [ 957.926561] LustreError: 37925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.945504] LustreError: 37925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 958.981076] LustreError: 37910:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345075 with bad export cookie 10744011470245211545 [ 958.997248] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.231422] LustreError: 40757:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 965.237107] LustreError: 40757:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 965.404545] Lustre: server umount lustre-MDT0001 complete [ 973.504883] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 975.928180] Key type lgssc unregistered [ 976.154791] LNet: 41291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.159789] LNetError: 41291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.170679] LNet: Removed LNI 192.168.204.143@tcp [ 976.945701] Key type .llcrypt unregistered [ 976.948487] Key type ._llcrypt unregistered [ 996.059223] Key type ._llcrypt registered [ 996.060674] Key type .llcrypt registered [ 996.207645] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 997.082862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 997.117184] alg: No test for adler32 (adler32-zlib) [ 998.179816] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 998.381762] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1000.024205] Key type lgssc registered [ 1001.074760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.946396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.960473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.337190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.765814] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1018.860359] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.889684] LustreError: 42662:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1020.563280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1020.881901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.808471] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1030.197926] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.597984] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1038.927278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1039.228472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.473919] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1048.279552] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1054.581919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1059.814297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.815950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1059.837808] Lustre: Skipped 1 previous similar message [ 1059.847674] Lustre: Skipped 1 previous similar message [ 1064.311880] LustreError: 44909:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1064.418291] Lustre: server umount lustre-MDT0000 complete [ 1064.929054] LustreError: 43377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.958259] LustreError: 43377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1067.424427] LustreError: 42644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345183 with bad export cookie 15406058238026386385 [ 1067.426990] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.434032] LustreError: 42644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1070.048833] LustreError: 43831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1070.049477] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.066446] LustreError: 43831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1070.082818] Lustre: Skipped 1 previous similar message [ 1070.086409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1070.092524] Lustre: Skipped 1 previous similar message [ 1072.102989] LustreError: 42657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.106812] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1072.116020] LustreError: 42657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1072.126672] Lustre: Skipped 1 previous similar message [ 1073.603326] LustreError: 45113:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1073.615403] LustreError: 45113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1073.785996] Lustre: server umount lustre-MDT0001 complete [ 1102.304794] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.347985] LustreError: 45336:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1102.351314] LustreError: 45336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1102.450963] Lustre: server umount lustre-OST0000 complete [ 1114.812958] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1117.033709] Key type lgssc unregistered [ 1117.317456] LNet: 46175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1117.342409] LNetError: 46175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.356974] LNet: Removed LNI 192.168.204.143@tcp [ 1118.300706] Key type .llcrypt unregistered [ 1118.302046] Key type ._llcrypt unregistered [ 1141.328479] Key type ._llcrypt registered [ 1141.330072] Key type .llcrypt registered [ 1141.442631] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1142.510506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1142.555267] alg: No test for adler32 (adler32-zlib) [ 1143.631765] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1143.832721] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1145.504230] Key type lgssc registered [ 1146.411553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1152.194638] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1152.214988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1167.713357] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1172.832248] LustreError: 47541:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 1199.456189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.455490] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1207.618861] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1374.412857] LustreError: 48248:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1374.512779] Lustre: server umount lustre-OST0000 complete [ 1388.982496] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1391.227854] Key type lgssc unregistered [ 1391.555907] LNet: 49088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.574529] LNetError: 49088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.614229] LNet: Removed LNI 192.168.204.143@tcp [ 1393.442839] Key type .llcrypt unregistered [ 1393.445476] Key type ._llcrypt unregistered [ 1415.046161] Key type ._llcrypt registered [ 1415.047431] Key type .llcrypt registered [ 1415.132601] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1415.629806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1415.655811] alg: No test for adler32 (adler32-zlib) [ 1416.670934] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1416.879636] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1418.536357] Key type lgssc registered [ 1419.583264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1426.926667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1426.965225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.368558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.867199] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1438.693585] LustreError: 50502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.713556] LustreError: 50502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1439.090112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1439.358131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.313478] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1447.073414] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1450.017453] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1455.003688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1455.271423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.014538] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1465.476534] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1469.922180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.928516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.938455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.658330] LustreError: 52726:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1473.766629] Lustre: server umount lustre-OST0000 complete [ 1479.141523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.170525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.980285] LustreError: 52928:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1482.990277] LustreError: 52928:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1483.116741] Lustre: server umount lustre-MDT0000 complete [ 1484.258172] LustreError: 50483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.285406] LustreError: 50483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1486.401949] LustreError: 52929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345602 with bad export cookie 13013772020444662542 [ 1486.409322] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.412813] LustreError: 52929:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1486.561920] LustreError: 53130:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1486.565542] LustreError: 53130:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1486.710782] Lustre: server umount lustre-MDT0001 complete [ 1496.742032] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1499.466915] Key type lgssc unregistered [ 1499.776520] LNet: 53663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.779406] LNetError: 53663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.841399] LNet: Removed LNI 192.168.204.143@tcp [ 1501.657588] Key type .llcrypt unregistered [ 1501.659318] Key type ._llcrypt unregistered [ 1523.888110] Key type ._llcrypt registered [ 1523.889810] Key type .llcrypt registered [ 1524.001947] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1524.700769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1524.722442] alg: No test for adler32 (adler32-zlib) [ 1525.848807] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1526.161262] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1527.848163] Key type lgssc registered [ 1528.783480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.179140] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.225099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1551.906687] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1557.024225] LustreError: 55028:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 1583.778294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.065119] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1594.847210] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1601.431564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.125580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.227295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1605.429403] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1607.137612] LustreError: 55054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.145487] LustreError: 55054:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1612.261150] LustreError: 55054:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.026711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1613.442135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.059651] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1622.854823] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.655599] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1628.643308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1628.645609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.670072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.156635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.174023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.179769] Lustre: Skipped 1 previous similar message [ 1634.276053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1634.282178] Lustre: Skipped 1 previous similar message [ 1634.638311] LustreError: 57306:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1634.751355] Lustre: server umount lustre-OST0000 complete [ 1659.362149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1659.367776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.419237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.863753] LustreError: 57683:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1661.869996] LustreError: 57683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1662.020636] Lustre: server umount lustre-MDT0000 complete [ 1663.977132] LustreError: 57432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.998514] LustreError: 57432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1665.427517] LustreError: 55037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345781 with bad export cookie 17260092458654578172 [ 1665.430450] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1665.443044] LustreError: 55037:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1671.674665] LustreError: 57886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1671.680166] LustreError: 57886:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1671.861250] Lustre: server umount lustre-MDT0001 complete [ 1681.324289] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1683.750563] Key type lgssc unregistered [ 1683.961697] LNet: 58418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.968251] LNetError: 58418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.985801] LNet: Removed LNI 192.168.204.143@tcp [ 1684.831915] Key type .llcrypt unregistered [ 1684.833960] Key type ._llcrypt unregistered [ 1706.219237] Key type ._llcrypt registered [ 1706.220911] Key type .llcrypt registered [ 1706.337382] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1707.189430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1707.264361] alg: No test for adler32 (adler32-zlib) [ 1708.304268] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1708.620962] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1710.296184] Key type lgssc registered [ 1711.284871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.460394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.489143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.924915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.971833] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1730.531632] LustreError: 59811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.541788] LustreError: 59811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1732.881594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1733.160382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1736.792618] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1742.162554] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1746.013935] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1752.224346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.498550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.209432] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1763.918845] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1764.816543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1779.173097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1779.187061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1779.208252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.200195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.201399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.201499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.635178] LustreError: 62084:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1784.716824] Lustre: server umount lustre-OST0000 complete [ 1789.412357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1789.425272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.435030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.444479] Lustre: Skipped 1 previous similar message [ 1794.210798] LustreError: 62285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1794.213533] LustreError: 62285:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1794.288201] Lustre: server umount lustre-MDT0000 complete [ 1795.553942] LustreError: 60991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.564418] LustreError: 60991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1797.102135] LustreError: 59791:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345913 with bad export cookie 8189165207137905931 [ 1797.108469] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.201472] LustreError: 62486:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1797.205462] LustreError: 62486:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1797.327573] Lustre: server umount lustre-MDT0001 complete [ 1805.273546] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1807.956968] Key type lgssc unregistered [ 1808.299763] LNet: 63018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.316483] LNetError: 63018:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.344727] LNet: Removed LNI 192.168.204.143@tcp [ 1809.307574] Key type .llcrypt unregistered [ 1809.308805] Key type ._llcrypt unregistered [ 1840.228717] Key type ._llcrypt registered [ 1840.230349] Key type .llcrypt registered [ 1840.305178] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1840.787475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1840.810829] alg: No test for adler32 (adler32-zlib) [ 1841.792669] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1841.967213] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1843.608157] Key type lgssc registered [ 1844.566550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.414887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.427606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.713366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.576747] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1861.283123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1861.520816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1864.306764] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1869.019914] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1872.472887] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1879.182031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1879.595772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.828908] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1888.946002] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1897.025313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1904.842226] Lustre: Failing over lustre-MDT0000 [ 1904.991027] LustreError: 66968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1905.117336] Lustre: server umount lustre-MDT0000 complete [ 1905.822644] LustreError: 64700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.844773] LustreError: 64700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1907.176151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.178953] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.189955] Lustre: Skipped 2 previous similar messages [ 1907.213180] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1910.920457] LustreError: 64698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.162429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1913.384830] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.548969] Lustre: *** cfs_fail_loc=135, val=0*** [ 1913.634515] LustreError: 67477:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1913.640229] LustreError: 67477:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.143@tcp: cfg command failed: rc = -2 [ 1913.645485] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1913.645485] [ 1913.655173] LustreError: MGC192.168.204.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1913.678192] LustreError: 67465:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1913.695894] LustreError: 67465:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1913.707694] LustreError: 67465:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1913.750605] Lustre: server umount lustre-MDT0000 complete [ 1913.752302] LustreError: 67465:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1916.033508] LustreError: 64699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.051252] LustreError: 64699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1919.656306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.767590] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.001920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.046685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1921.160630] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1923.157892] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1925.104735] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1925.115051] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1925.150657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1930.210523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.213607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1930.225710] Lustre: Skipped 1 previous similar message [ 1931.968095] LustreError: 68286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1931.985552] LustreError: 68286:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1932.145373] Lustre: server umount lustre-OST0000 complete [ 1935.337991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.339652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.340056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.340060] Lustre: Skipped 1 previous similar message [ 1935.349524] Lustre: Skipped 1 previous similar message [ 1940.449337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.453086] Lustre: Skipped 2 previous similar messages [ 1941.566154] LustreError: 68486:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1941.569275] LustreError: 68486:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1941.664158] Lustre: server umount lustre-MDT0000 complete [ 1945.273841] LustreError: 64684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346061 with bad export cookie 2702286479776075913 [ 1945.282057] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.285948] LustreError: 64684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1945.461358] LustreError: 68688:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1945.466523] LustreError: 68688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1945.668115] Lustre: server umount lustre-MDT0001 complete [ 1955.291648] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1957.802450] Key type lgssc unregistered [ 1958.102474] LNet: 69221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.117073] LNetError: 69221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.132796] LNet: Removed LNI 192.168.204.143@tcp [ 1958.851780] Key type .llcrypt unregistered [ 1958.853295] Key type ._llcrypt unregistered [ 1979.414467] Key type ._llcrypt registered [ 1979.416316] Key type .llcrypt registered [ 1979.499893] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1980.383308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.471434] alg: No test for adler32 (adler32-zlib) [ 1981.548966] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1981.738281] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1983.376296] Key type lgssc registered [ 1984.293764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.315710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1991.336448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.690429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.146435] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2001.892874] LustreError: 70611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.908204] LustreError: 70611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2003.351812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2003.655452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2007.043256] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2012.233829] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2015.657240] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2021.805485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2022.091132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.500154] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2032.569232] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2037.547397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2043.281508] Lustre: Failing over lustre-MDT0000 [ 2043.404072] LustreError: 72859:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2043.496538] Lustre: server umount lustre-MDT0000 complete [ 2044.585627] LustreError: 72587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.911602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2044.914534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.973168] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.980923] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.998255] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2052.088992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2052.170992] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2052.295036] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2052.434849] LustreError: 73368:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2052.439685] LustreError: 73368:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.143@tcp: cfg command failed: rc = -9 [ 2052.448527] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2052.448527] [ 2052.454058] LustreError: MGC192.168.204.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2052.480348] LustreError: 73356:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2052.494966] LustreError: 73356:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2052.508220] LustreError: 73356:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2052.532517] Lustre: server umount lustre-MDT0000 complete [ 2052.534517] LustreError: 73356:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2053.091275] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2053.112503] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2058.219963] LustreError: 72769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.234476] LustreError: 72769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2058.740913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.915298] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2059.139056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.200274] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2059.912273] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2063.177097] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2064.371908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2064.413618] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2064.456973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2069.987062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2069.995077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.011799] Lustre: Skipped 1 previous similar message [ 2070.024304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2071.808495] LustreError: 74179:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2071.812932] LustreError: 74179:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2071.901557] Lustre: server umount lustre-OST0000 complete [ 2079.719347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.726164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2082.539960] LustreError: 74380:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2082.546738] LustreError: 74380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2082.598924] Lustre: server umount lustre-MDT0000 complete [ 2084.838337] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.859912] LustreError: 70612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2085.833256] LustreError: 70593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346202 with bad export cookie 16256118365717463445 [ 2085.835845] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.843581] LustreError: 70593:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.122847] LustreError: 74583:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2096.143299] LustreError: 74583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2096.269962] Lustre: server umount lustre-MDT0001 complete [ 2107.712058] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2110.480755] Key type lgssc unregistered [ 2110.812398] LNet: 75117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.819559] LNetError: 75117:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.848860] LNet: Removed LNI 192.168.204.143@tcp [ 2111.637677] Key type .llcrypt unregistered [ 2111.640142] Key type ._llcrypt unregistered [ 2123.307772] Key type ._llcrypt registered [ 2123.309795] Key type .llcrypt registered [ 2123.441183] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2123.791602] Key type .llcrypt unregistered [ 2123.793279] Key type ._llcrypt unregistered [ 2145.564632] Key type ._llcrypt registered [ 2145.565944] Key type .llcrypt registered [ 2145.746692] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2146.475753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2146.560197] alg: No test for adler32 (adler32-zlib) [ 2147.726324] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2148.044370] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2149.840212] Key type lgssc registered [ 2151.422585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2159.279368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2159.295447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.862823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.609581] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2171.364105] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2171.385494] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2172.031669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2172.353455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.721680] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2180.998484] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2184.604832] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2191.068753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2191.429194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.918285] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2202.033321] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2205.748558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2215.907421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.910858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2215.913273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.915802] Lustre: Skipped 1 previous similar message [ 2220.989172] LustreError: 79248:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2221.034878] LustreError: 78579:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.044792] LustreError: 78579:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2221.131080] Lustre: server umount lustre-OST0000 complete [ 2226.148993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.157266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.163348] Lustre: Skipped 1 previous similar message [ 2226.173677] Lustre: Skipped 2 previous similar messages [ 2230.706143] LustreError: 79450:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2230.709892] LustreError: 79450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2230.805617] Lustre: server umount lustre-MDT0000 complete [ 2231.267815] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.292723] LustreError: 76981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2234.518061] LustreError: 76963:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346350 with bad export cookie 17001008502565614856 [ 2234.536672] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.660990] LustreError: 79651:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2234.665044] LustreError: 79651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2234.839108] Lustre: server umount lustre-MDT0001 complete [ 2245.257798] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2248.365859] Key type lgssc unregistered [ 2248.704838] LNet: 80183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.718882] LNetError: 80183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.768216] LNet: Removed LNI 192.168.204.143@tcp [ 2250.579190] Key type .llcrypt unregistered [ 2250.580501] Key type ._llcrypt unregistered [ 2272.538746] Key type ._llcrypt registered [ 2272.540459] Key type .llcrypt registered [ 2272.676679] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2273.910751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.948471] alg: No test for adler32 (adler32-zlib) [ 2275.228902] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2275.530203] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2277.272148] Key type lgssc registered [ 2278.428837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.397069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.442275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.152359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.631154] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2299.369020] LustreError: 81598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.386402] LustreError: 81598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2300.436906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2300.867500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.337336] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2310.119462] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2313.822205] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2319.626878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2320.072413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.555909] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2330.870676] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2334.451044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2341.859911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.863647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.870797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.929310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2344.940759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.941161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.963839] Lustre: Skipped 1 previous similar message [ 2347.063264] LustreError: 83844:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2347.130708] Lustre: server umount lustre-OST0000 complete [ 2352.100387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2352.103636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.114320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.170863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.172935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.398197] LustreError: 84046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2356.400918] LustreError: 84046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2356.483705] Lustre: server umount lustre-MDT0000 complete [ 2360.025675] LustreError: 81579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346476 with bad export cookie 1035950942995134887 [ 2360.031957] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.049136] LustreError: 81579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2360.181611] LustreError: 84248:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2360.200241] LustreError: 84248:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2360.421684] Lustre: server umount lustre-MDT0001 complete [ 2369.013425] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2371.656530] Key type lgssc unregistered [ 2371.917519] LNet: 84780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.935304] LNetError: 84780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.947680] LNet: Removed LNI 192.168.204.143@tcp [ 2372.717215] Key type .llcrypt unregistered [ 2372.721221] Key type ._llcrypt unregistered [ 2393.000836] Key type ._llcrypt registered [ 2393.002369] Key type .llcrypt registered [ 2393.075293] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2394.004061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2394.088602] alg: No test for adler32 (adler32-zlib) [ 2395.340172] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2395.660953] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2397.384230] Key type lgssc registered [ 2398.493338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2406.380533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2406.404807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.005576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.405169] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2418.148371] LustreError: 86154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.160237] LustreError: 86154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2420.073788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2420.510514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2424.540951] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2429.431455] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2433.168784] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2440.532680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2440.943429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.821555] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2451.055589] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2453.243195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2467.823170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2467.827103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.834066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.416117] LustreError: 88431:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2468.572556] Lustre: server umount lustre-OST0000 complete [ 2473.963831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.971398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.981114] Lustre: Skipped 1 previous similar message [ 2478.709261] LustreError: 88631:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2478.721761] LustreError: 88631:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2478.839409] Lustre: server umount lustre-MDT0000 complete [ 2479.075599] LustreError: 86155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.097609] LustreError: 86155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2482.367686] LustreError: 86134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346598 with bad export cookie 7410961778724593475 [ 2482.384967] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.386904] LustreError: 86134:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2491.889759] LustreError: 88833:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2491.900627] LustreError: 88833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2492.076132] Lustre: server umount lustre-MDT0001 complete [ 2502.322912] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2504.580117] Key type lgssc unregistered [ 2504.785190] LNet: 89367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.791589] LNetError: 89367:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.803304] LNet: Removed LNI 192.168.204.143@tcp [ 2505.419782] Key type .llcrypt unregistered [ 2505.422621] Key type ._llcrypt unregistered [ 2526.112993] Key type ._llcrypt registered [ 2526.114763] Key type .llcrypt registered [ 2526.242062] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2527.521751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2527.604610] alg: No test for adler32 (adler32-zlib) [ 2528.680280] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2528.936732] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2530.608196] Key type lgssc registered [ 2531.946617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.711252] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.738503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2555.296390] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2560.417828] LustreError: 90733:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 2587.040181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2591.883878] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2596.102805] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2600.682303] LustreError: 91590:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2600.767742] Lustre: server umount lustre-OST0000 complete [ 2608.485400] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:32:03 (1763346723) [ 2614.436103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.813982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.676424] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2619.878901] LustreError: 92259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.894690] LustreError: 92259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2625.003540] LustreError: 92259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.638645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2627.898265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2631.458697] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2636.340554] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.392794] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2645.738923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2646.084735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.672988] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2657.389988] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2660.408774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2673.634628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2673.651556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.676229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.688289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2675.690921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2675.695955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.252161] LustreError: 94665:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2677.254103] LustreError: 94665:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2677.335649] Lustre: server umount lustre-OST0000 complete [ 2680.802733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.809849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.812602] Lustre: Skipped 1 previous similar message [ 2685.925767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.932038] Lustre: Skipped 1 previous similar message [ 2686.417675] LustreError: 94866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2686.420748] LustreError: 94866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2686.479948] Lustre: server umount lustre-MDT0000 complete [ 2690.094717] LustreError: 92241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346806 with bad export cookie 6195179509728966816 [ 2690.104257] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2690.111904] LustreError: 92241:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2690.282184] LustreError: 95069:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2690.294786] LustreError: 95069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2690.582179] Lustre: server umount lustre-MDT0001 complete [ 2701.755577] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2704.362733] Key type lgssc unregistered [ 2704.648301] LNet: 95601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2704.660515] LNetError: 95601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2704.693540] LNet: Removed LNI 192.168.204.143@tcp [ 2705.740842] Key type .llcrypt unregistered [ 2705.748769] Key type ._llcrypt unregistered [ 2726.643280] Key type ._llcrypt registered [ 2726.646440] Key type .llcrypt registered [ 2726.746072] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2727.487737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2727.521907] alg: No test for adler32 (adler32-zlib) [ 2728.510651] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2728.758508] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2730.432201] Key type lgssc registered [ 2731.231759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2737.541233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2737.568294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.984820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.509950] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2749.409946] LustreError: 96976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.432789] LustreError: 96976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2749.694802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2749.973889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2753.063758] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2757.617267] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2760.730564] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.170877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2767.460921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.725449] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2778.008787] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2783.893108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2791.396079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2791.403297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.411844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2794.472409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.483165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.484026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2794.497279] Lustre: Skipped 1 previous similar message [ 2797.241107] LustreError: 99245:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2797.455524] Lustre: server umount lustre-OST0000 complete [ 2804.712314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.720138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.724598] Lustre: Skipped 1 previous similar message [ 2807.407248] LustreError: 99446:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2807.413806] LustreError: 99446:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2807.473599] Lustre: server umount lustre-MDT0000 complete [ 2809.829083] LustreError: 96975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.883520] LustreError: 96975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2811.015654] LustreError: 96956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346927 with bad export cookie 4692661340045843264 [ 2811.026157] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.031654] LustreError: 96956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2811.190466] LustreError: 99648:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2811.193381] LustreError: 99648:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2811.391989] Lustre: server umount lustre-MDT0001 complete [ 2820.422833] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2822.678926] Key type lgssc unregistered [ 2822.922803] LNet: 100180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.929966] LNetError: 100180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.946978] LNet: Removed LNI 192.168.204.143@tcp [ 2823.608885] Key type .llcrypt unregistered [ 2823.611079] Key type ._llcrypt unregistered [ 2839.595346] Key type ._llcrypt registered [ 2839.601363] Key type .llcrypt registered [ 2839.729726] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2840.875913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2840.948221] alg: No test for adler32 (adler32-zlib) [ 2841.988654] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2842.254686] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2843.920198] Key type lgssc registered [ 2844.960401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2852.004433] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2852.017361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2867.552510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2872.672237] LustreError: 101401:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 2899.429290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.924421] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2909.417916] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2914.763944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.988853] LustreError: MGC192.168.204.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2914.996096] LustreError: 102323:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2915.001851] LustreError: 102323:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2915.011925] LustreError: 102323:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2915.017148] LustreError: 101410:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347031 with bad export cookie 12672584096085788690 [ 2915.031954] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2915.103915] Lustre: server umount lustre-MDT0000 complete [ 2915.107582] LustreError: 102323:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2940.451107] LustreError: 102754:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2940.532942] Lustre: server umount lustre-OST0000 complete [ 2948.698099] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 2957.231132] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2966.024711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2971.312084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2976.245521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2982.297360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2989.108145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2989.167420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2989.346443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2989.373245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2989.434631] Lustre: lustre-MDT0000: new disk, initializing [ 2989.511309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.530836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2992.598535] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3003.106500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3003.196802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3003.292620] Lustre: 105094:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3003.340696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3003.345744] Lustre: Skipped 1 previous similar message [ 3003.414469] Lustre: lustre-MDT0001: new disk, initializing [ 3003.542465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3003.622565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3003.634889] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3007.555572] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3015.849545] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3019.673102] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3026.246242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3026.334227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3026.576979] Lustre: lustre-OST0000: new disk, initializing [ 3026.584955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3026.678489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.614794] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3032.648692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3032.671257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3032.854894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3038.143536] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3041.207719] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3041.399557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3044.649940] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3044.891923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3048.422478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3048.427875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3048.439305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3048.443890] Lustre: Skipped 1 previous similar message [ 3052.499783] LustreError: 108030:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3052.516829] LustreError: 108030:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3052.655526] Lustre: server umount lustre-OST0000 complete [ 3058.657539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3058.664440] Lustre: Skipped 1 previous similar message [ 3058.669585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.985223] LustreError: 108234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3061.989748] LustreError: 108234:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3062.073266] Lustre: server umount lustre-MDT0000 complete [ 3064.830963] LustreError: 105087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347181 with bad export cookie 12672584096085788788 [ 3064.838023] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.840797] LustreError: 105087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3064.958529] LustreError: 108435:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3064.961552] LustreError: 108435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3065.129169] Lustre: server umount lustre-MDT0001 complete [ 3072.438401] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:39:47 (1763347187) [ 3075.135774] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3088.524311] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 3095.914889] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3103.216773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3108.005470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3112.379704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3116.791387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.002894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3122.053971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.197948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3122.219651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3122.282440] Lustre: lustre-MDT0000: new disk, initializing [ 3122.352761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.371520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3125.471255] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3133.809481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3133.885102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3133.978667] Lustre: 111570:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3134.032334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3134.035467] Lustre: Skipped 1 previous similar message [ 3134.095909] Lustre: lustre-MDT0001: new disk, initializing [ 3134.173897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.200858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3134.206059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3136.743207] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3142.785279] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3145.751231] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3150.750502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3150.804143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3151.010340] Lustre: lustre-OST0000: new disk, initializing [ 3151.024773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3151.105843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.120418] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3161.134130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3161.148818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3161.231564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3161.290901] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3163.989781] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3164.103920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3166.871836] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3167.046099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3171.297552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3171.301039] LustreError: Skipped 1 previous similar message [ 3171.303070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.312959] Lustre: Skipped 1 previous similar message [ 3171.320658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3171.323592] Lustre: Skipped 1 previous similar message [ 3174.649286] LustreError: 114509:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3174.652471] LustreError: 114509:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3174.718539] Lustre: server umount lustre-OST0000 complete [ 3180.516151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3180.519943] LustreError: Skipped 1 previous similar message [ 3180.521821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3180.528044] Lustre: Skipped 1 previous similar message [ 3180.530913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.537406] Lustre: Skipped 1 previous similar message [ 3183.268726] LustreError: 114710:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3183.276193] LustreError: 114710:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3183.341336] Lustre: server umount lustre-MDT0000 complete [ 3185.841646] LustreError: 112349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347302 with bad export cookie 12672584096085789663 [ 3185.851060] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3185.854078] LustreError: 112349:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3186.089592] Lustre: server umount lustre-MDT0001 complete [ 3191.637369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.980101] LustreError: 115390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.992715] LustreError: 115390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3192.045093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.572409] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3197.413025] LustreError: 115390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.432614] LustreError: 115390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3200.911208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3203.948642] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3208.009891] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3210.993425] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3216.670262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3221.415075] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3225.197688] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.929807] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3236.326777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3236.332225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.336387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3236.338653] Lustre: Skipped 2 previous similar messages [ 3236.347202] Lustre: Skipped 3 previous similar messages [ 3239.682770] LustreError: 117731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3239.689361] LustreError: 117731:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3239.764763] Lustre: server umount lustre-OST0000 complete [ 3246.562940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3246.564060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3246.576295] Lustre: Skipped 1 previous similar message [ 3246.581828] Lustre: Skipped 1 previous similar message [ 3249.596282] Lustre: server umount lustre-MDT0000 complete [ 3251.682958] LustreError: 115384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3251.692131] LustreError: 115384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3252.328954] LustreError: 115369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347368 with bad export cookie 12672584096085790552 [ 3252.334281] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.335355] LustreError: 115369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3252.530506] Lustre: server umount lustre-MDT0001 complete [ 3260.363858] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3262.305940] Key type lgssc unregistered [ 3262.519681] LNet: 118668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3262.525353] LNetError: 118668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3262.540716] LNet: Removed LNI 192.168.204.143@tcp [ 3263.067656] Key type .llcrypt unregistered [ 3263.069499] Key type ._llcrypt unregistered [ 3276.060069] Key type ._llcrypt registered [ 3276.061447] Key type .llcrypt registered [ 3276.145602] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 3286.790153] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3287.558482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3287.572168] alg: No test for adler32 (adler32-zlib) [ 3288.496117] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 3288.641597] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3290.265196] Key type lgssc registered [ 3291.151125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3296.680932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3301.001738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3305.230805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3309.063725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.168982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3313.200723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3313.210819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.332624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3314.357592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3314.415974] Lustre: lustre-MDT0000: new disk, initializing [ 3314.474828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.487123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3316.777258] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3323.666146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3323.713956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3323.784502] Lustre: 121773:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3323.842386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3323.845610] Lustre: Skipped 1 previous similar message [ 3323.888752] Lustre: lustre-MDT0001: new disk, initializing [ 3323.938505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3323.967133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3323.980535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3326.350428] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3331.208807] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3333.699031] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3338.805390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3338.859471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3339.030273] Lustre: lustre-OST0000: new disk, initializing [ 3339.032973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3339.092327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.919618] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3344.915303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3344.925674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3344.992830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3347.585592] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3349.973677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3350.101208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3352.740180] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3352.976839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3355.108573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3355.108883] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.112378] LustreError: Skipped 1 previous similar message [ 3355.113645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3355.120402] Lustre: Skipped 1 previous similar message [ 3360.234537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3360.236766] Lustre: Skipped 1 previous similar message [ 3360.517785] LustreError: 124710:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3360.648774] Lustre: server umount lustre-OST0000 complete [ 3364.835617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3364.845775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.863606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.478464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3370.488772] Lustre: Skipped 3 previous similar messages [ 3375.587665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.596110] Lustre: Skipped 1 previous similar message [ 3378.145332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3378.197705] LustreError: 124912:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3378.200528] LustreError: 124912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3378.243667] Lustre: server umount lustre-MDT0000 complete [ 3380.513352] LustreError: 124914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347496 with bad export cookie 233396961893009770 [ 3380.513938] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3380.520144] LustreError: 124914:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3380.640698] LustreError: 125117:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3380.643574] LustreError: 125117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3380.727559] Lustre: server umount lustre-MDT0001 complete [ 3385.779249] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:45:01 (1763347501) [ 3389.844396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.159219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.464323] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3395.556495] LustreError: 125786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.565510] LustreError: 125786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3398.637914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3398.879063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3401.826586] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3405.672535] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.004580] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3410.611515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.613305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3410.618400] Lustre: Skipped 2 previous similar messages [ 3416.789287] LustreError: 127129:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3416.791852] LustreError: 127129:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3416.858702] Lustre: server umount lustre-MDT0000 complete [ 3419.925816] LustreError: 125768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347536 with bad export cookie 233396961893010659 [ 3419.929814] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.941238] LustreError: 125768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3420.374417] Lustre: server umount lustre-MDT0001 complete [ 3428.514861] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:45:43 (1763347543) [ 3434.704717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3450.208874] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3455.331595] LustreError: 127977:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 3481.056272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.182477] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3487.358281] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3488.608737] LustreError: 128641:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3488.611653] LustreError: 128641:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3488.692161] Lustre: server umount lustre-OST0000 complete [ 3494.545803] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:46:50 (1763347610) [ 3499.545187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.874444] LustreError: 129312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.939839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.017519] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3505.126476] LustreError: 129312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.589790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3512.841552] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3516.948222] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3519.761559] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3524.671030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3529.035083] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3533.483164] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3545.570894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.573358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3545.574558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3545.576939] Lustre: Skipped 1 previous similar message [ 3545.579980] Lustre: Skipped 3 previous similar messages [ 3549.208942] LustreError: 131556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3549.212130] LustreError: 131556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3549.264813] Lustre: server umount lustre-MDT0000 complete [ 3550.695324] LustreError: 130098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.707438] LustreError: 130098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3551.709929] LustreError: 129293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347668 with bad export cookie 233396961893011310 [ 3551.712195] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3551.723539] LustreError: 129293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3551.979704] Lustre: server umount lustre-MDT0001 complete [ 3565.173382] Lustre: server umount lustre-OST0000 complete [ 3572.562441] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:48:07 (1763347687) [ 3577.993328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.391641] LustreError: 132632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.466156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3578.469932] Lustre: Skipped 2 previous similar messages [ 3581.731537] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3588.167079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3590.916319] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3594.571859] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3597.062142] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3602.355546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3604.658473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3606.822063] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3610.662811] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3613.514879] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3613.681238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3616.685721] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3616.848378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3618.115230] LustreError: 135372:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3618.117775] LustreError: 135372:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3618.176107] Lustre: server umount lustre-OST0000 complete [ 3623.394600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3623.396066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3623.405227] Lustre: Skipped 2 previous similar messages [ 3623.411396] Lustre: Skipped 3 previous similar messages [ 3634.656193] Lustre: 119987:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763347734/real 1763347734] req@00000000e45c498c x1849003767595136/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763347750 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3634.673595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.168100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3635.323365] Lustre: server umount lustre-MDT0000 complete [ 3638.109030] LustreError: 132612:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347754 with bad export cookie 233396961893012374 [ 3638.130629] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3638.352875] Lustre: server umount lustre-MDT0001 complete [ 3643.791478] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:49:19 (1763347759) [ 3647.594866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3663.136927] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3668.256439] LustreError: 136419:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 3693.856313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.859113] Lustre: Skipped 2 previous similar messages [ 3698.453808] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3702.413733] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3707.176341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.441391] LustreError: 136446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.451407] LustreError: 136446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3707.522159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3709.983991] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3716.261126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3720.240516] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3724.333993] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3726.936151] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3729.577412] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3729.728772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3732.127591] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3732.268225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3737.061052] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.062031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3737.062555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.062563] Lustre: Skipped 6 previous similar messages [ 3737.067895] Lustre: Skipped 1 previous similar message [ 3739.688761] LustreError: 139193:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3739.695407] LustreError: 139193:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3739.803892] Lustre: server umount lustre-MDT0000 complete [ 3742.179463] LustreError: 137344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.192821] LustreError: 137344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3743.322625] LustreError: 136428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347859 with bad export cookie 233396961893013403 [ 3743.328910] LustreError: 136428:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3743.331488] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3743.591317] Lustre: server umount lustre-MDT0001 complete [ 3755.991843] Lustre: server umount lustre-OST0000 complete [ 3763.153037] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:51:18 (1763347878) [ 3768.417474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3783.904700] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3789.024406] LustreError: 140243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 3821.087141] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3826.291519] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.505656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3833.111478] LustreError: 140270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.176868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.182475] Lustre: Skipped 3 previous similar messages [ 3833.240537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3836.462859] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3843.369889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.245701] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3849.953532] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3853.076407] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3859.266357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3859.309338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3859.424443] Lustre: lustre-OST0001: new disk, initializing [ 3859.435384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3864.771200] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3869.209921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3869.220603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3869.289647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3870.225725] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3872.961653] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3873.088235] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3876.216977] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3876.475952] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3879.396434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3879.400218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.402165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3879.405313] Lustre: Skipped 2 previous similar messages [ 3879.411420] Lustre: Skipped 4 previous similar messages [ 3892.199912] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3892.278980] LustreError: 143973:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3892.281609] LustreError: 143973:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3892.410395] Lustre: server umount lustre-OST0000 complete [ 3899.879620] LustreError: 140258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.884477] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.903504] LustreError: 140258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3899.911867] LustreError: Skipped 2 previous similar messages [ 3902.639812] Lustre: server umount lustre-OST0001 complete [ 3921.376370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3921.493297] Lustre: server umount lustre-MDT0000 complete [ 3924.085893] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348040 with bad export cookie 233396961893014348 [ 3924.087393] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3924.095030] LustreError: 140250:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3924.319656] Lustre: server umount lustre-MDT0001 complete [ 3928.306530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3931.182103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3934.098816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.680166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3940.723694] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:54:16 (1763348056) [ 3941.742217] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3942.819572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3943.944469] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:54:19 (1763348059) [ 3948.290182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.475905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3948.501922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3951.581801] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3957.341446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3957.403784] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3957.433509] Lustre: 146219:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3960.372947] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3964.089602] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3966.649661] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3971.712255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3971.832827] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3975.754173] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3978.331700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3979.690461] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3982.642629] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3982.774562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3985.807704] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3985.951795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3988.453345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.464206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.470576] Lustre: Skipped 4 previous similar messages [ 3988.473108] LustreError: Skipped 1 previous similar message [ 3993.604511] Lustre: server umount lustre-OST0000 complete [ 4012.222839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4016.198285] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4019.388333] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4022.488316] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4027.817179] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.827114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.827629] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4027.839022] Lustre: Skipped 1 previous similar message [ 4028.884343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4031.323595] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4031.431651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4033.448401] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e6688fbb800.ost_server_uuid 50 [ 4034.384204] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e6688fbb800.ost_server_uuid in FULL state after 0 sec [ 4034.516650] LustreError: 146226:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.43@tcp) returned error from blocking AST (req@000000005e362e06 x1849003767718400 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000af6f1a0b/0x33d314cdd49e491 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.43@tcp remote: 0x58b8ef10db522fc3 expref: 5 pid: 147353 timeout: 4134 lvb_type: 0 [ 4034.537075] LustreError: lustre-MDT0000: A client on nid 192.168.204.43@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4034.541976] LustreError: 146215:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.43@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000af6f1a0b/0x33d314cdd49e491 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.43@tcp remote: 0x58b8ef10db522fc3 expref: 6 pid: 147353 timeout: 0 lvb_type: 0 [ 4038.114146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.117755] LustreError: Skipped 1 previous similar message [ 4042.877567] LustreError: 150813:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4042.880312] LustreError: 150813:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 4042.938652] Lustre: server umount lustre-OST0000 complete [ 4051.274812] Lustre: server umount lustre-MDT0000 complete [ 4053.446374] LustreError: 146210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348169 with bad export cookie 233396961893015524 [ 4053.452690] LustreError: 146210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4053.454806] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4059.669543] Lustre: server umount lustre-MDT0001 complete [ 4065.877548] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4067.569433] Key type lgssc unregistered [ 4067.775064] LNet: 151752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4067.783359] LNetError: 151752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4067.805637] LNet: Removed LNI 192.168.204.143@tcp [ 4068.391222] Key type .llcrypt unregistered [ 4068.393634] Key type ._llcrypt unregistered [ 4083.867603] Key type ._llcrypt registered [ 4083.869026] Key type .llcrypt registered [ 4083.923510] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4084.403310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4084.420391] alg: No test for adler32 (adler32-zlib) [ 4085.349127] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4085.480418] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4087.112758] Key type lgssc registered [ 4087.710870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4092.443625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4092.463645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.847510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.907434] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4103.025240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4103.340390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4105.885446] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4109.541797] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4112.143600] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4116.354553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4116.568231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.667793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4119.688972] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4122.730212] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4131.694237] Lustre: Failing over lustre-MDT0000 [ 4131.815048] LustreError: 155372:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4131.890353] Lustre: server umount lustre-MDT0000 complete [ 4133.347765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.352478] LustreError: 153123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4133.353281] Lustre: Skipped 2 previous similar messages [ 4133.363604] LustreError: 153123:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4138.469746] LustreError: 154285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4138.479583] LustreError: 154285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4143.436727] LustreError: 153103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.43@tcp arrived at 1763348259 with bad export cookie 6081610892617073444 [ 4143.444460] LustreError: 153103:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4143.586033] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.593571] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4147.665849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.752648] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4147.903853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.941993] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4150.328876] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4151.940719] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4151.947090] Lustre: lustre-MDT0000: Denying connection for new client 48566d53-77af-417b-8d43-45a0c47bf875 (at 192.168.204.43@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 4153.320284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4163.556240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4163.566021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4163.570765] Lustre: Skipped 1 previous similar message [ 4165.497167] LustreError: 156305:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4165.499353] LustreError: 156305:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4165.568526] Lustre: server umount lustre-OST0000 complete [ 4168.008518] LustreError: 156506:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4168.017203] Lustre: 155872:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4168.023571] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4168.048404] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4168.059428] LustreError: 155872:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fca3d64a x1849004603250048/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4168.078480] LustreError: 155872:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4168.089064] LustreError: 155872:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4168.097868] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 0 recovered and 2 were evicted. [ 4168.225944] LustreError: 156506:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4168.231844] LustreError: 156506:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4168.293184] Lustre: server umount lustre-MDT0000 complete [ 4168.673481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.679267] Lustre: Skipped 1 previous similar message [ 4168.682185] LustreError: 155488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.906738] LustreError: 154744:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348287 with bad export cookie 6081610892617073549 [ 4170.908151] LustreError: 152433:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e2d26618 x1849004603251200/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4170.909525] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4170.912302] LustreError: 154744:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4177.060388] LustreError: 156708:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4177.065499] LustreError: 156708:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4177.191386] Lustre: server umount lustre-MDT0001 complete [ 4184.045533] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4185.749683] Key type lgssc unregistered [ 4185.947607] LNet: 157240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4185.957394] LNetError: 157240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4185.969691] LNet: Removed LNI 192.168.204.143@tcp [ 4186.417955] Key type .llcrypt unregistered [ 4186.420303] Key type ._llcrypt unregistered [ 4200.858424] Key type ._llcrypt registered [ 4200.859734] Key type .llcrypt registered [ 4200.958240] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4201.484818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4201.559430] alg: No test for adler32 (adler32-zlib) [ 4202.506410] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4202.660498] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4204.312188] Key type lgssc registered [ 4205.040370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4211.291345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4211.328432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.805640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.698557] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4222.948018] LustreError: 158613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.958946] LustreError: 158613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4223.105116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4223.349668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.662478] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4231.103843] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4234.107699] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4239.679465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4239.941444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.756890] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4247.777982] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4258.416516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4263.909709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4263.914961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.923335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4263.935110] Lustre: Skipped 1 previous similar message [ 4263.944148] Lustre: Skipped 1 previous similar message [ 4264.981640] LustreError: 160881:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4265.127275] Lustre: server umount lustre-OST0000 complete [ 4269.028886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.035948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4269.038024] Lustre: Skipped 1 previous similar message [ 4274.148885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.158977] Lustre: Skipped 2 previous similar messages [ 4274.562660] LustreError: 161083:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4274.575534] LustreError: 161083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4274.705534] Lustre: server umount lustre-MDT0000 complete [ 4277.954884] LustreError: 158594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348394 with bad export cookie 17085889888770267330 [ 4277.961756] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4277.962514] LustreError: 158594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4278.098735] LustreError: 161284:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4278.107096] LustreError: 161284:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4278.371136] Lustre: server umount lustre-MDT0001 complete [ 4285.674369] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4287.748470] Key type lgssc unregistered [ 4287.986437] LNet: 161816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4287.996926] LNetError: 161816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4289.063093] LNet: Removed LNI 192.168.204.143@tcp [ 4289.724897] Key type .llcrypt unregistered [ 4289.726615] Key type ._llcrypt unregistered [ 4313.003434] Key type ._llcrypt registered [ 4313.005091] Key type .llcrypt registered [ 4313.067597] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4313.648830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4313.661932] alg: No test for adler32 (adler32-zlib) [ 4314.623254] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4314.814466] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4316.440743] Key type lgssc registered [ 4317.186521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4321.903907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4321.918986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.187498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.193049] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4331.346076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4331.706854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4334.336291] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4337.555935] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4339.993731] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4344.484925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4344.741127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.826537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4348.503604] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4351.390290] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4361.186655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4361.187548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4361.191983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4361.198813] Lustre: Skipped 1 previous similar message [ 4366.305643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4366.309068] Lustre: Skipped 1 previous similar message [ 4367.158121] LustreError: 165735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4367.229919] Lustre: server umount lustre-OST0000 complete [ 4371.430766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4371.432526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4371.448556] Lustre: Skipped 1 previous similar message [ 4371.458189] Lustre: Skipped 1 previous similar message [ 4375.543178] LustreError: 165937:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4375.552744] LustreError: 165937:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4375.644240] Lustre: server umount lustre-MDT0000 complete [ 4376.546148] LustreError: 163480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4376.557240] LustreError: 163480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4377.909833] LustreError: 163459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348494 with bad export cookie 12889541222721344008 [ 4377.915643] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.917794] LustreError: 163459:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4377.994814] LustreError: 166140:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4377.997773] LustreError: 166140:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4378.107578] Lustre: server umount lustre-MDT0001 complete [ 4384.768902] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4386.893141] Key type lgssc unregistered [ 4387.089491] LNet: 166673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.098432] LNetError: 166673:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4387.111509] LNet: Removed LNI 192.168.204.143@tcp [ 4387.667568] Key type .llcrypt unregistered [ 4387.670422] Key type ._llcrypt unregistered [ 4400.701501] Key type ._llcrypt registered [ 4400.705902] Key type .llcrypt registered [ 4400.766273] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4401.204685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4401.261366] alg: No test for adler32 (adler32-zlib) [ 4402.162911] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4402.286713] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4403.920129] Key type lgssc registered [ 4404.779317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4414.125941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4414.138120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.426806] Lustre: *** cfs_fail_loc=135, val=0*** [ 4415.505182] LustreError: 168154:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4415.508937] LustreError: 168154:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.143@tcp: cfg command failed: rc = -2 [ 4415.515305] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4415.515305] [ 4415.520133] LustreError: MGC192.168.204.143@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4415.532890] LustreError: 168115:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4415.541720] LustreError: 168115:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4415.550856] LustreError: 168115:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4415.563228] LustreError: 168115:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4415.626458] Lustre: server umount lustre-MDT0000 complete [ 4415.629871] LustreError: 168115:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4421.070779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.436492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.217361] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4430.820211] LustreError: 168527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.843853] LustreError: 168527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4430.870867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4431.197827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4433.985811] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4437.916283] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4441.258394] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.688941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4446.690749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.700339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.702497] Lustre: Skipped 1 previous similar message [ 4448.865542] LustreError: 169871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4448.869037] LustreError: 169871:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4448.928539] Lustre: server umount lustre-MDT0000 complete [ 4451.510829] LustreError: 168506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348567 with bad export cookie 10624448598827736415 [ 4451.514922] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.516298] LustreError: 168506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4451.631829] LustreError: 170073:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4451.633926] LustreError: 170073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4451.765763] Lustre: server umount lustre-MDT0001 complete [ 4457.936754] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4459.700982] Key type lgssc unregistered [ 4459.917064] LNet: 170607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4459.922581] LNetError: 170607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4459.941947] LNet: Removed LNI 192.168.204.143@tcp [ 4460.428443] Key type .llcrypt unregistered [ 4460.431168] Key type ._llcrypt unregistered [ 4473.018247] Key type ._llcrypt registered [ 4473.019691] Key type .llcrypt registered [ 4473.083460] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4473.245431] Key type .llcrypt unregistered [ 4473.247168] Key type ._llcrypt unregistered [ 4483.820546] Key type ._llcrypt registered [ 4483.822925] Key type .llcrypt registered [ 4483.908223] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4484.482948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4484.499439] alg: No test for adler32 (adler32-zlib) [ 4485.447808] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4485.569801] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4487.184210] Key type lgssc registered [ 4487.907973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4492.883661] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4492.897499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4508.385443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4513.505244] LustreError: 172675:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 4540.256817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4544.203913] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4548.054460] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4552.835995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4553.173554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4553.232136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4555.847546] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4560.366640] LustreError: 173599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.375782] LustreError: 173599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4562.387349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.643946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.506278] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4570.047293] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4573.150842] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4575.436639] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4588.517337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.520857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.528300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4591.739469] LustreError: 175395:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4591.819265] Lustre: server umount lustre-OST0000 complete [ 4598.755022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4598.755377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4598.758466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4598.758474] Lustre: Skipped 1 previous similar message [ 4598.775268] Lustre: Skipped 2 previous similar messages [ 4600.720333] LustreError: 175597:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4600.723187] LustreError: 175597:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4600.771748] Lustre: server umount lustre-MDT0000 complete [ 4602.962530] LustreError: 172682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348719 with bad export cookie 10836910200622326143 [ 4602.965112] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4602.971785] LustreError: 172682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4603.088648] LustreError: 175800:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4603.091683] LustreError: 175800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4603.206611] Lustre: server umount lustre-MDT0001 complete [ 4610.313253] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4612.897874] Key type lgssc unregistered [ 4613.193468] LNet: 176332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4613.209449] LNetError: 176332:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4614.250636] LNet: Removed LNI 192.168.204.143@tcp [ 4614.911523] Key type .llcrypt unregistered [ 4614.913219] Key type ._llcrypt unregistered [ 4635.892419] Key type ._llcrypt registered [ 4635.894121] Key type .llcrypt registered [ 4636.004608] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4636.887523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4636.898273] alg: No test for adler32 (adler32-zlib) [ 4637.907231] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4638.094898] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4639.752306] Key type lgssc registered [ 4640.676205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4647.246359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4647.260942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.574766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.706138] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4658.155104] LustreError: 177725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.171112] LustreError: 177725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4659.228802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4659.491945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4662.293438] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4665.890681] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4668.305189] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4672.918189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4672.996438] LustreError: 179323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.166633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.358766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4677.013882] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4681.125359] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4686.000330] Lustre: Failing over lustre-MDT0000 [ 4686.097684] LustreError: 180020:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4686.208082] Lustre: server umount lustre-MDT0000 complete [ 4689.055691] LustreError: 177720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.895818] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.901732] Lustre: Skipped 1 previous similar message [ 4694.153452] LustreError: 177720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.173355] LustreError: 177720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4699.268814] LustreError: 177719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4699.290930] LustreError: 177719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4703.258263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.457498] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.738829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.777448] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4704.396412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4707.065068] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4708.848190] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4708.889345] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4708.947722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4711.135512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4718.237879] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 4724.198798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.203928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4724.212765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.226474] Lustre: Skipped 1 previous similar message [ 4728.632104] LustreError: 181302:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4728.635283] LustreError: 181302:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4728.753974] Lustre: server umount lustre-OST0000 complete [ 4734.433471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4734.439333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.444348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4734.453171] Lustre: Skipped 2 previous similar messages [ 4734.464805] Lustre: Skipped 1 previous similar message [ 4738.476222] LustreError: 181508:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4738.485369] LustreError: 181508:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4738.564854] Lustre: server umount lustre-MDT0000 complete [ 4739.553774] LustreError: 177725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.572105] LustreError: 177725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4741.436720] LustreError: 177704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348857 with bad export cookie 16250354456462842645 [ 4741.440695] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4741.454094] LustreError: 177704:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4741.700686] LustreError: 181709:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4741.707236] LustreError: 181709:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4741.902834] Lustre: server umount lustre-MDT0001 complete [ 4749.629398] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4751.446922] Key type lgssc unregistered [ 4751.665988] LNet: 182240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4751.675509] LNetError: 182240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4751.686721] LNet: Removed LNI 192.168.204.143@tcp [ 4752.245206] Key type .llcrypt unregistered [ 4752.251243] Key type ._llcrypt unregistered [ 4771.146472] Key type ._llcrypt registered [ 4771.150596] Key type .llcrypt registered [ 4771.267747] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4772.069105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4772.115664] alg: No test for adler32 (adler32-zlib) [ 4773.182940] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4773.359432] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4775.016415] Key type lgssc registered [ 4775.987728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4783.383344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4783.398034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.751786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.337964] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4795.367179] LustreError: 183612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.409823] LustreError: 183612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4797.162710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4797.520241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4802.224622] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4807.728158] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4811.013441] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4817.732415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4818.150320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.063329] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4828.047255] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4830.451818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4831.908197] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4834.945922] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4845.125607] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4856.294815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.297788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4856.298150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4856.301144] Lustre: Skipped 1 previous similar message [ 4861.410372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4861.413279] Lustre: Skipped 1 previous similar message [ 4861.621518] LustreError: 186029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4861.730432] Lustre: server umount lustre-OST0000 complete [ 4866.531116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.534651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.547427] Lustre: Skipped 1 previous similar message [ 4871.288541] LustreError: 186230:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4871.295450] LustreError: 186230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4871.373659] Lustre: server umount lustre-MDT0000 complete [ 4871.649706] LustreError: 184327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.665721] LustreError: 184327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4874.540966] LustreError: 183595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348990 with bad export cookie 2203603736705137091 [ 4874.541932] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4874.577886] LustreError: 183595:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4874.727600] LustreError: 186432:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4874.750132] LustreError: 186432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4874.915251] Lustre: server umount lustre-MDT0001 complete [ 4883.591566] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4886.067741] Key type lgssc unregistered [ 4886.315973] LNet: 186963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.323665] LNetError: 186963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4887.344876] LNet: Removed LNI 192.168.204.143@tcp [ 4888.212626] Key type .llcrypt unregistered [ 4888.214277] Key type ._llcrypt unregistered [ 4909.424125] Key type ._llcrypt registered [ 4909.425514] Key type .llcrypt registered [ 4909.552898] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4910.754667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4910.796400] alg: No test for adler32 (adler32-zlib) [ 4911.899467] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 4912.047498] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4913.688232] Key type lgssc registered [ 4915.289715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4924.137491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4924.152350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.917960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.190930] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4936.167472] LustreError: 188337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.205508] LustreError: 188337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4937.401950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4937.649698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4940.397352] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4944.430590] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4947.755305] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4952.905320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4953.148321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.125867] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4960.938444] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4971.578401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4976.430871] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4985.960079] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4994.919680] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5002.774223] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5013.476994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5013.480229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.494634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5017.572233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5017.572290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.595056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5022.693209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5022.708252] Lustre: Skipped 2 previous similar messages [ 5027.296267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5027.333116] LustreError: 191996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5027.463224] Lustre: server umount lustre-OST0000 complete [ 5032.931833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.953690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.439181] LustreError: 192199:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5037.445594] LustreError: 192199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5037.583418] Lustre: server umount lustre-MDT0000 complete [ 5038.059644] LustreError: 188337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.073046] LustreError: 188337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5041.145295] LustreError: 188319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349157 with bad export cookie 6389633554913975435 [ 5041.159281] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5041.179631] LustreError: 188319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5041.341796] LustreError: 192401:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5041.350936] LustreError: 192401:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5041.663598] Lustre: server umount lustre-MDT0001 complete [ 5051.893906] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5054.019883] Key type lgssc unregistered [ 5054.213603] LNet: 192933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5054.218278] LNetError: 192933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5054.227595] LNet: Removed LNI 192.168.204.143@tcp [ 5054.825241] Key type .llcrypt unregistered [ 5054.827538] Key type ._llcrypt unregistered [ 5073.325425] Key type ._llcrypt registered [ 5073.326878] Key type .llcrypt registered [ 5073.429055] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5074.138598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5074.212944] alg: No test for adler32 (adler32-zlib) [ 5075.292698] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 5075.602630] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5077.256171] Key type lgssc registered [ 5078.004846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5083.569708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5083.581554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5084.914119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.278238] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5094.203664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5094.516182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5098.511305] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5104.326138] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5107.993417] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5115.123538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5115.463949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.365318] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5125.777082] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5129.772346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5145.059047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5145.063868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.071398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5145.088062] Lustre: Skipped 1 previous similar message [ 5145.095676] Lustre: Skipped 1 previous similar message [ 5148.360090] LustreError: 196707:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5148.511482] Lustre: server umount lustre-OST0000 complete [ 5155.298766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.301864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.312324] Lustre: Skipped 1 previous similar message [ 5155.319289] Lustre: Skipped 1 previous similar message [ 5160.418828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.537572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5165.542710] Lustre: Skipped 2 previous similar messages [ 5166.048659] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5166.123043] LustreError: 196909:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5166.126477] LustreError: 196909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5166.190844] Lustre: server umount lustre-MDT0000 complete [ 5169.277891] LustreError: 194287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349285 with bad export cookie 14381675970676521850 [ 5169.291258] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5169.405458] LustreError: 197113:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5169.411299] LustreError: 197113:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5169.567933] Lustre: server umount lustre-MDT0001 complete [ 5178.223482] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5180.540123] Key type lgssc unregistered [ 5180.787173] LNet: 197643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5180.793547] LNetError: 197643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5180.811500] LNet: Removed LNI 192.168.204.143@tcp [ 5181.444696] Key type .llcrypt unregistered [ 5181.447430] Key type ._llcrypt unregistered [ 5195.201780] Key type ._llcrypt registered [ 5195.203400] Key type .llcrypt registered [ 5195.282633] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5195.861904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5195.873520] alg: No test for adler32 (adler32-zlib) [ 5196.770882] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 5196.930496] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5198.546500] Key type lgssc registered [ 5199.452532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5205.878386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5205.905034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.521488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.943777] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5216.740152] LustreError: 198869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.747521] LustreError: 198869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5217.100452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5217.324126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.099425] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5223.910581] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5226.308544] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5231.610418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5231.962880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.018803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5236.422041] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5240.106755] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5252.579481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.582783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5252.588144] Lustre: Skipped 1 previous similar message [ 5252.597546] Lustre: Skipped 1 previous similar message [ 5257.702772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.708267] Lustre: Skipped 1 previous similar message [ 5262.820321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5262.823213] Lustre: Skipped 1 previous similar message [ 5263.328150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5263.366147] LustreError: 201264:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5263.464411] Lustre: server umount lustre-OST0000 complete [ 5267.938986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.946427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5267.951871] Lustre: Skipped 1 previous similar message [ 5272.164068] LustreError: 201467:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5272.167055] LustreError: 201467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5272.227334] Lustre: server umount lustre-MDT0000 complete [ 5273.057236] LustreError: 199581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.072426] LustreError: 199581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5275.025366] LustreError: 201470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349391 with bad export cookie 11662162208855071835 [ 5275.025647] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.031983] LustreError: 201470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5275.137422] LustreError: 201671:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5275.140100] LustreError: 201671:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5275.254597] Lustre: server umount lustre-MDT0001 complete [ 5282.941931] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5285.059455] Key type lgssc unregistered [ 5285.295992] LNet: 202203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5285.311411] LNetError: 202203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5285.331749] LNet: Removed LNI 192.168.204.143@tcp [ 5286.073756] Key type .llcrypt unregistered [ 5286.075317] Key type ._llcrypt unregistered [ 5300.729114] Key type ._llcrypt registered [ 5300.730818] Key type .llcrypt registered [ 5300.854810] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5301.430683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5301.444854] alg: No test for adler32 (adler32-zlib) [ 5302.403433] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 5302.606570] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5304.240170] Key type lgssc registered [ 5305.121499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5312.011749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5312.040935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.725672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5317.422808] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5323.236191] LustreError: 203431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.271332] LustreError: 203431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5324.795550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5325.161533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5329.145823] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5334.862560] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5338.964933] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5345.724904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5346.048353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.187626] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5354.347941] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5358.325543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5366.273690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5366.394658] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5366.582750] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.748979] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5376.665631] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5378.137743] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5389.690671] Lustre: Permanently deactivating lustre-OST0001 [ 5389.708956] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5398.558649] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5404.771730] Lustre: Permanently reactivating lustre-OST0001 [ 5404.792978] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5404.796956] Lustre: Skipped 2 previous similar messages [ 5405.880985] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.902379] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5405.913915] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5405.925270] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5405.963084] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5416.422561] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5416.427489] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.439026] Lustre: Skipped 1 previous similar message [ 5416.443171] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5416.450022] Lustre: Skipped 1 previous similar message [ 5417.446889] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5417.462580] Lustre: Skipped 1 previous similar message [ 5420.030533] LustreError: 206900:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5420.081459] Lustre: server umount lustre-OST0001 complete [ 5422.566713] LustreError: 205032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.579964] LustreError: 205032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5425.121928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5425.132056] LustreError: Skipped 1 previous similar message [ 5425.134076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5425.144123] Lustre: Skipped 1 previous similar message [ 5425.156555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.178914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.182750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.192847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.685620] LustreError: 206813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.695394] LustreError: 206813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5429.377251] LustreError: 207103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5429.388654] LustreError: 207103:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5429.550645] Lustre: server umount lustre-OST0000 complete [ 5432.804169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.807660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5432.808064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5432.811278] Lustre: Skipped 1 previous similar message [ 5432.813416] Lustre: Skipped 1 previous similar message [ 5438.685212] LustreError: 207305:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5438.688243] LustreError: 207305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5438.794478] Lustre: server umount lustre-MDT0000 complete [ 5441.793081] LustreError: 203412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349558 with bad export cookie 7640013443166625190 [ 5441.801454] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.805798] LustreError: 203412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5441.942530] LustreError: 207509:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5441.948583] LustreError: 207509:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5442.068933] Lustre: server umount lustre-MDT0001 complete [ 5448.622329] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5450.720219] Key type lgssc unregistered [ 5450.965755] LNet: 208040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5450.970657] LNetError: 208040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5450.980729] LNet: Removed LNI 192.168.204.143@tcp [ 5451.573598] Key type .llcrypt unregistered [ 5451.575788] Key type ._llcrypt unregistered [ 5474.985686] Key type ._llcrypt registered [ 5474.987376] Key type .llcrypt registered [ 5475.060292] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 5485.776957] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5486.528290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5486.543507] alg: No test for adler32 (adler32-zlib) [ 5487.501739] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 5487.647159] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5489.288282] Key type lgssc registered [ 5490.152723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5497.394744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5502.033714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5506.658727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5511.057674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5516.265383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5516.304294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5516.322472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5517.625913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5517.669346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5517.758901] Lustre: lustre-MDT0000: new disk, initializing [ 5517.856809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5517.871860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5521.861238] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5530.354836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5530.414289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5530.468812] Lustre: 211964:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5530.517978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5530.520675] Lustre: Skipped 1 previous similar message [ 5530.570280] Lustre: lustre-MDT0001: new disk, initializing [ 5530.662786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5530.692713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5530.702043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5534.662735] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5541.211553] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5544.528892] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5550.545097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5550.624444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5550.873108] Lustre: lustre-OST0000: new disk, initializing [ 5550.876463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5550.951779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5555.906234] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5557.849994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5557.864471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5558.007405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5563.374527] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5566.846653] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5567.038682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5569.754866] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5569.905301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5573.604751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.608351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5573.608447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5573.623129] Lustre: Skipped 1 previous similar message [ 5577.341233] LustreError: 214901:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5577.465317] Lustre: server umount lustre-OST0000 complete [ 5581.797185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5581.801690] LustreError: Skipped 1 previous similar message [ 5581.810240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.825115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.827879] Lustre: Skipped 1 previous similar message [ 5583.845096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.846638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5583.863973] Lustre: Skipped 1 previous similar message [ 5586.687425] LustreError: 215102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5586.704099] LustreError: 215102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5586.782941] Lustre: server umount lustre-MDT0000 complete [ 5589.635148] LustreError: 211956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349705 with bad export cookie 515560943153644785 [ 5589.641312] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5589.642755] LustreError: 211956:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5589.749213] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5589.759094] LustreError: 215303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5589.872864] Lustre: server umount lustre-MDT0001 complete [ 5597.317656] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 22:21:52 (1763349712) [ 5603.151740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.661130] LustreError: 215975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5603.668339] LustreError: 215975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5603.736467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.787950] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5608.944205] LustreError: 215975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.053579] LustreError: 215975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.493050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5615.901959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5619.369666] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5624.970357] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5627.711754] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5633.944320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5634.198539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.457644] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5645.361798] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5652.250255] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 5655.622851] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5660.968732] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 5668.839496] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5678.696287] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5699.436420] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 5699.440219] Lustre: Skipped 2 previous similar messages [ 5737.429674] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5737.434892] Lustre: Skipped 4 previous similar messages [ 5804.558218] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 5804.562073] Lustre: Skipped 8 previous similar messages [ 5810.660655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5810.667847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.685946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.734348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5815.684184] LustreError: 219449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5815.700970] LustreError: 219449:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5815.810678] Lustre: server umount lustre-OST0000 complete [ 5820.899519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5820.915336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.935462] Lustre: Skipped 1 previous similar message [ 5820.946657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.949415] Lustre: Skipped 2 previous similar messages [ 5825.189765] LustreError: 219651:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5825.197039] LustreError: 219651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5825.298100] Lustre: server umount lustre-MDT0000 complete [ 5828.637238] LustreError: 218281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349944 with bad export cookie 515560943153645667 [ 5828.641714] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5829.077741] Lustre: server umount lustre-MDT0001 complete [ 5838.687201] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5841.731864] Key type lgssc unregistered [ 5841.983601] LNet: 220387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.005628] LNetError: 220387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5842.039457] LNet: Removed LNI 192.168.204.143@tcp [ 5842.901795] Key type .llcrypt unregistered [ 5842.905916] Key type ._llcrypt unregistered [ 5865.114622] Key type ._llcrypt registered [ 5865.115976] Key type .llcrypt registered [ 5865.222888] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5866.000481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5866.024149] alg: No test for adler32 (adler32-zlib) [ 5867.021846] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 5867.230774] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5868.888242] Key type lgssc registered [ 5869.693983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5875.860065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5875.883245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.302215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5880.414677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5887.458805] LustreError: 221778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.483695] LustreError: 221778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5887.697906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5888.064970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.212424] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5896.013714] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.084836] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5905.026426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5905.375871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.160373] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5914.223651] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5937.121807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5937.125615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.134183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5937.144868] Lustre: Skipped 1 previous similar message [ 5937.155742] Lustre: Skipped 1 previous similar message [ 5942.244167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.253160] Lustre: Skipped 1 previous similar message [ 5942.972392] LustreError: 224196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5943.105141] Lustre: server umount lustre-OST0000 complete [ 5947.363448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.369223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5947.377800] Lustre: Skipped 1 previous similar message [ 5947.408566] Lustre: Skipped 1 previous similar message [ 5952.481774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5952.489978] Lustre: Skipped 1 previous similar message [ 5952.621745] LustreError: 224401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5952.623711] LustreError: 224401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5952.706124] Lustre: server umount lustre-MDT0000 complete [ 5955.914731] LustreError: 224047:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350072 with bad export cookie 14238371610914565214 [ 5955.921532] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5956.043049] LustreError: 224602:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5956.054081] LustreError: 224602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5956.276121] Lustre: server umount lustre-MDT0001 complete [ 5967.134339] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5969.741457] Key type lgssc unregistered [ 5970.009305] LNet: 225134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5970.013791] LNetError: 225134:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5970.026087] LNet: Removed LNI 192.168.204.143@tcp [ 5970.728715] Key type .llcrypt unregistered [ 5970.730638] Key type ._llcrypt unregistered [ 5989.744402] Key type ._llcrypt registered [ 5989.746093] Key type .llcrypt registered [ 5989.867129] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5990.229732] Key type .llcrypt unregistered [ 5990.237142] Key type ._llcrypt unregistered [ 6014.270178] Key type ._llcrypt registered [ 6014.274032] Key type .llcrypt registered [ 6014.345698] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 6028.005050] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 6028.826679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6028.861894] alg: No test for adler32 (adler32-zlib) [ 6029.844126] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 6030.025193] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 6031.656494] Key type lgssc registered [ 6032.720838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6040.149127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6045.453982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6050.907918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6057.434271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6059.104934] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5841 sec ========== 22:29:34 (1763350174) [ 6060.890821] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:29:35 (1763350175) === [ 6064.458850] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:29:39 (1763350179) === [ 6100.242907] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 6102.585686] Key type lgssc unregistered [ 6102.874504] LNet: 231244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6102.883662] LNetError: 231244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6102.902886] LNet: Removed LNI 192.168.204.143@tcp [ 6103.635890] Key type .llcrypt unregistered [ 6103.637762] Key type ._llcrypt unregistered