[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460572673 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009108] LSM: Security Framework initializing [ 0.010044] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.019684] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023678] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026103] *** VALIDATE tmpfs *** [ 0.028327] *** VALIDATE proc *** [ 0.029169] *** VALIDATE cgroup *** [ 0.030008] *** VALIDATE cgroup2 *** [ 0.031218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032137] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034030] Spectre V2 : User space: Vulnerable [ 0.035010] Speculative Store Bypass: Vulnerable [ 0.038358] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.041000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041618] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042020] ... version: 2 [ 0.042920] ... bit width: 48 [ 0.043014] ... generic registers: 4 [ 0.044010] ... value mask: 0000ffffffffffff [ 0.045010] ... max period: 00007fffffffffff [ 0.046008] ... fixed-purpose events: 3 [ 0.047009] ... event mask: 000000070000000f [ 0.048249] rcu: Hierarchical SRCU implementation. [ 0.050283] smp: Bringing up secondary CPUs ... [ 0.051512] x86: Booting SMP configuration: [ 0.052023] .... node #0, CPUs: #1 #2 #3 [ 0.055501] smp: Brought up 1 node, 4 CPUs [ 0.057024] smpboot: Max logical packages: 1 [ 0.058014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142034] node 0 deferred pages initialised in 83ms [ 0.144087] devtmpfs: initialized [ 0.145207] x86/mm: Memory block size: 128MB [ 0.148760] gcov: version magic: 0x41383552 [ 0.151038] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154241] pinctrl core: initialized pinctrl subsystem [ 0.155144] [ 0.155610] ************************************************************* [ 0.158011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160010] ** ** [ 0.161005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163010] ** ** [ 0.164000] ** This means that this kernel is built to expose internal ** [ 0.165011] ** IOMMU data structures, which may compromise security on ** [ 0.166007] ** your system. ** [ 0.168009] ** ** [ 0.169009] ** If you see this message and you are not debugging the ** [ 0.171011] ** kernel, report this immediately to your vendor! ** [ 0.173010] ** ** [ 0.175011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177011] ************************************************************* [ 0.178583] NET: Registered protocol family 16 [ 0.180403] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187040] cpuidle: using governor menu [ 0.188441] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190471] PCI: Using configuration type 1 for base access [ 0.192132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204043] cryptd: max_cpu_qlen set to 1000 [ 0.206275] ACPI: Added _OSI(Module Device) [ 0.208125] ACPI: Added _OSI(Processor Device) [ 0.210014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215482] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222142] ACPI: Interpreter enabled [ 0.223047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225010] ACPI: Using IOAPIC for interrupt routing [ 0.226000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227345] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236609] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248432] acpiphp: Slot [2] registered [ 0.250126] acpiphp: Slot [5] registered [ 0.251106] acpiphp: Slot [6] registered [ 0.252104] acpiphp: Slot [7] registered [ 0.253080] acpiphp: Slot [8] registered [ 0.254025] acpiphp: Slot [9] registered [ 0.255091] acpiphp: Slot [10] registered [ 0.256101] acpiphp: Slot [3] registered [ 0.257071] acpiphp: Slot [4] registered [ 0.259079] acpiphp: Slot [11] registered [ 0.260111] acpiphp: Slot [12] registered [ 0.261082] acpiphp: Slot [13] registered [ 0.262077] acpiphp: Slot [14] registered [ 0.264074] acpiphp: Slot [15] registered [ 0.265057] acpiphp: Slot [16] registered [ 0.266069] acpiphp: Slot [17] registered [ 0.267098] acpiphp: Slot [18] registered [ 0.268068] acpiphp: Slot [19] registered [ 0.269079] acpiphp: Slot [20] registered [ 0.270068] acpiphp: Slot [21] registered [ 0.272098] acpiphp: Slot [22] registered [ 0.273078] acpiphp: Slot [23] registered [ 0.274089] acpiphp: Slot [24] registered [ 0.275076] acpiphp: Slot [25] registered [ 0.277079] acpiphp: Slot [26] registered [ 0.278076] acpiphp: Slot [27] registered [ 0.279060] acpiphp: Slot [28] registered [ 0.281101] acpiphp: Slot [29] registered [ 0.282083] acpiphp: Slot [30] registered [ 0.284087] acpiphp: Slot [31] registered [ 0.285060] PCI host bridge to bus 0000:00 [ 0.286017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.288016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297132] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298857] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301155] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309032] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313038] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.317010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320365] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.329808] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348990] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354609] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.368025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.383016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.392902] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.397012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.406029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.421025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.432703] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.445026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.451020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.471021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.479773] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.486014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.493000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.511015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.522934] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.526016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.533019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.551021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.560675] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.568012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.573012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.590022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.598972] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.601857] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.604404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.606298] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.607171] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.612168] iommu: Default domain type: Passthrough [ 0.614546] SCSI subsystem initialized [ 0.616146] ACPI: bus type USB registered [ 0.617168] usbcore: registered new interface driver usbfs [ 0.619120] usbcore: registered new interface driver hub [ 0.620107] usbcore: registered new device driver usb [ 0.622195] pps_core: LinuxPPS API ver. 1 registered [ 0.624015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.627095] PTP clock support registered [ 0.629102] EDAC MC: Ver: 3.0.0 [ 0.631054] PCI: Using ACPI for IRQ routing [ 0.633233] NetLabel: Initializing [ 0.634010] NetLabel: domain hash size = 128 [ 0.636015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.638098] NetLabel: unlabeled traffic allowed by default [ 0.640161] vgaarb: loaded [ 0.641302] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.643012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.648004] clocksource: Switched to clocksource kvm-clock [ 0.747187] VFS: Disk quotas dquot_6.6.0 [ 0.748599] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.751223] *** VALIDATE ramfs *** [ 0.752199] *** VALIDATE hugetlbfs *** [ 0.753369] pnp: PnP ACPI init [ 0.755135] pnp: PnP ACPI: found 6 devices [ 0.770239] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.772642] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.774531] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.776644] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.779193] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.781331] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.783285] NET: Registered protocol family 2 [ 0.785530] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.790329] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.793459] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.798194] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.801432] TCP: Hash tables configured (established 65536 bind 65536) [ 0.804110] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.807346] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.809400] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.812203] NET: Registered protocol family 1 [ 0.814790] RPC: Registered named UNIX socket transport module. [ 0.817062] RPC: Registered udp transport module. [ 0.818517] RPC: Registered tcp transport module. [ 0.819874] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.822037] NET: Registered protocol family 44 [ 0.824776] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.827120] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.828617] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.831273] PCI: CLS 0 bytes, default 64 [ 0.832867] Unpacking initramfs... [ 2.215344] debug: unmapping init [mem 0xffff8cca3cc54000-0xffff8cca3ffbffff] [ 2.220362] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222268] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225681] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.702428] Initialise system trusted keyrings [ 2.704038] Key type blacklist registered [ 2.705762] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.715552] zbud: loaded [ 2.717831] *** VALIDATE nfs *** [ 2.719188] *** VALIDATE nfs4 *** [ 2.720715] pstore: using deflate compression [ 2.724066] Platform Keyring initialized [ 2.818446] NET: Registered protocol family 38 [ 2.820714] Key type asymmetric registered [ 2.822086] Asymmetric key parser 'x509' registered [ 2.823828] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.827219] io scheduler mq-deadline registered [ 2.828598] io scheduler kyber registered [ 2.830162] io scheduler bfq registered [ 2.831969] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.834517] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.836722] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.839358] ACPI: Power Button [PWRF] [ 2.844338] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.850361] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.866690] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.874041] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.894190] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.922565] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.952617] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.956856] Non-volatile memory driver v1.3 [ 2.958201] Linux agpgart interface v0.103 [ 2.988956] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.991706] vda: detected capacity change from 0 to 75083776 [ 3.006538] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.009242] vdb: detected capacity change from 0 to 1073741824 [ 3.023296] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.026083] vdc: detected capacity change from 0 to 2621440000 [ 3.039253] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.041745] vdd: detected capacity change from 0 to 2621440000 [ 3.057668] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.060296] vde: detected capacity change from 0 to 4294967296 [ 3.073662] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.076791] vdf: detected capacity change from 0 to 4294967296 [ 3.082882] libphy: Fixed MDIO Bus: probed [ 3.089274] usbcore: registered new interface driver usbserial_generic [ 3.091329] usbserial: USB Serial support registered for generic [ 3.093648] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.097516] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.099329] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.101816] mousedev: PS/2 mouse device common for all mice [ 3.104127] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.105536] rtc_cmos 00:05: RTC can wake from S4 [ 3.109801] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.110380] rtc_cmos 00:05: registered as rtc0 [ 3.114630] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.114809] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.120302] intel_pstate: CPU model not supported [ 3.123217] hid: raw HID events driver (C) Jiri Kosina [ 3.124865] usbcore: registered new interface driver usbhid [ 3.126605] usbhid: USB HID core driver [ 3.128163] drop_monitor: Initializing network drop monitor service [ 3.130449] Initializing XFRM netlink socket [ 3.132217] NET: Registered protocol family 10 [ 3.135193] Segment Routing with IPv6 [ 3.136419] NET: Registered protocol family 17 [ 3.138146] mpls_gso: MPLS GSO support [ 3.144170] RAS: Correctable Errors collector initialized. [ 3.146410] AVX version of gcm_enc/dec engaged. [ 3.148021] AES CTR mode by8 optimization enabled [ 3.219323] sched_clock: Marking stable (3219302091, 0)->(4033237883, -813935792) [ 3.221664] registered taskstats version 1 [ 3.223143] Loading compiled-in X.509 certificates [ 3.224665] zswap: loaded using pool lzo/zbud [ 3.245534] Key type big_key registered [ 3.257046] Key type encrypted registered [ 3.258570] ima: No TPM chip found, activating TPM-bypass! [ 3.260534] ima: Allocated hash algorithm: sha1 [ 3.262065] ima: No architecture policies found [ 3.263877] evm: Initialising EVM extended attributes: [ 3.265849] evm: security.selinux [ 3.267116] evm: security.ima [ 3.268235] evm: security.capability [ 3.269461] evm: HMAC attrs: 0x1 [ 3.271814] rtc_cmos 00:05: setting system clock to 2026-09-08 11:13:15 UTC (1788865995) [ 3.278362] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.281025] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 3.290129] Write protecting the kernel read-only data: 28672k [ 3.293166] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.296018] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 3.325489] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.333368] systemd[1]: Detected virtualization kvm. [ 3.335112] systemd[1]: Detected architecture x86-64. [ 3.337403] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.361371] systemd[1]: No hostname configured. [ 3.363246] systemd[1]: Set hostname to . [ 3.364929] random: systemd: uninitialized urandom read (16 bytes read) [ 3.366636] systemd[1]: Initializing machine ID from random generator. [ 3.498659] random: systemd: uninitialized urandom read (16 bytes read) [ 3.501179] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.507095] random: systemd: uninitialized urandom read (16 bytes read) [ 3.508804] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.512808] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. [ 4.053048] random: fast init done Starting dracut pre-udev hook... [ 4.099576] device-mapper: uevent: version 1.0.3 [ 4.101340] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.837433] virtio_net virtio0 ens2: renamed from eth0 [ 4.861419] scsi host0: ata_piix [ 4.868236] scsi host1: ata_piix [ 4.869521] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.871877] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.158477] dracut-initqueue[595]: RTNETLINK answers: File exists [ 9.784513] random: crng init done [ 9.785931] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.082376] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.185385] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.426179] SELinux: Disabled at runtime. [ 11.481341] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.488346] systemd[1]: Detected virtualization kvm. [ 11.490066] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.973333] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.976907] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.982769] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.990330] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.995143] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.002771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.006991] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.081079] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.452870] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.836240] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.874189] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.978459] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.990117] EDAC sbridge: Ver: 1.1.2 [ 14.274350] Key type dns_resolver registered [ 14.567960] NFS: Registering the id_resolver key type [ 14.569686] Key type id_resolver registered [ 14.570853] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg650-server login: [ 41.005844] libcfs: loading out-of-tree module taints kernel. [ 41.030765] Key type ._llcrypt registered [ 41.032083] Key type .llcrypt registered [ 41.086381] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 50.331826] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 51.059470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.068531] alg: No test for adler32 (adler32-zlib) [ 52.197923] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 52.611498] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 54.256608] Key type lgssc registered [ 55.056433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 63.952019] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 80.460435] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 85.410781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 85.420030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 86.511365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 86.529722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 86.571861] Lustre: lustre-MDT0000: new disk, initializing [ 86.605604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 86.614516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 88.120815] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 93.681817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.726743] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 93.743265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 93.745459] Lustre: Skipped 1 previous similar message [ 93.785913] Lustre: lustre-MDT0001: new disk, initializing [ 93.809957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 93.821392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 93.826538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 95.270686] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.779569] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 101.356936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 101.456260] Lustre: lustre-OST0000: new disk, initializing [ 101.458834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 101.461552] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 101.480647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.646492] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.372949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.379469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.403745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 109.404709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 109.453872] Lustre: lustre-OST0001: new disk, initializing [ 109.457210] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.460411] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 109.483317] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.381224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 110.384819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 110.411104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 111.601774] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.746808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.447611] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 122.673379] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing check_logdir /tmp/testlogs/ [ 124.534378] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing yml_node [ 126.247779] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 127.304872] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 128.450773] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 129.168497] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 07:15:21 EDT 2026 [ 136.017908] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 136.620536] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 150.496512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 150.500056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 150.505735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.520529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 151.522346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.585574] Lustre: server umount lustre-MDT0000 complete [ 155.820694] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866148 with bad export cookie 16112675068272923810 [ 155.822154] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 155.825859] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 156.640603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.640987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 156.645090] Lustre: Skipped 3 previous similar messages [ 156.648563] Lustre: Skipped 4 previous similar messages [ 160.672784] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.675396] Lustre: Skipped 1 previous similar message [ 161.958367] Lustre: server umount lustre-MDT0001 complete [ 163.260568] Lustre: server umount lustre-OST0000 complete [ 164.586270] Lustre: server umount lustre-OST0001 complete [ 166.976966] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 170.393975] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 183.926760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.016357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.029925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.068467] Lustre: lustre-MDT0000: new disk, initializing [ 184.107850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.116542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.659801] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 190.312725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.363153] Lustre: 15054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.481729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.048918] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 196.289164] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 198.543769] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 201.335517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.466669] Lustre: lustre-OST0000: new disk, initializing [ 201.469289] Lustre: Skipped 1 previous similar message [ 201.472447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.475313] Lustre: Skipped 2 previous similar messages [ 201.478234] Lustre: 17023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.525283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.529107] Lustre: Skipped 1 previous similar message [ 202.957992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 202.962610] Lustre: Skipped 1 previous similar message [ 202.966132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.017722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.319182] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 209.004514] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 211.042599] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 211.163442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 213.022565] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 213.114569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 218.596403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 218.597542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 218.601860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 218.604722] LustreError: Skipped 1 previous similar message [ 218.609316] Lustre: Skipped 1 previous similar message [ 228.320104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 228.375873] Lustre: server umount lustre-OST0000 complete [ 231.392932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 231.396687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.402560] Lustre: Skipped 1 previous similar message [ 231.405145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 231.407971] Lustre: Skipped 2 previous similar messages [ 244.192112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 244.252454] Lustre: server umount lustre-MDT0000 complete [ 245.680650] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866237 with bad export cookie 16112675068272925329 [ 245.681393] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.686484] LustreError: 15049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 245.821709] Lustre: server umount lustre-MDT0001 complete [ 247.173554] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:17:19 (1788866239) [ 249.688854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 249.866148] LustreError: 19288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 249.873812] LustreError: 19288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 249.891978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.407310] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 254.563469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 256.185735] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 258.853819] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 260.769401] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 263.075796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.179810] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 267.825673] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 284.640912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 284.643651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.647360] Lustre: Skipped 1 previous similar message [ 284.649619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 284.651875] Lustre: Skipped 6 previous similar messages [ 295.392117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 295.453225] Lustre: server umount lustre-OST0000 complete [ 301.026035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.026811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 301.036561] Lustre: Skipped 2 previous similar messages [ 301.043326] LustreError: Skipped 1 previous similar message [ 311.266531] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 311.363126] Lustre: server umount lustre-MDT0000 complete [ 312.977545] LustreError: 19269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866305 with bad export cookie 16112675068272926211 [ 312.980118] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.985717] LustreError: 19269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 313.132592] Lustre: server umount lustre-MDT0001 complete [ 317.531749] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 318.811193] Key type lgssc unregistered [ 318.956528] LNet: 22839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.960565] LNetError: 22839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.969898] LNet: Removed LNI 192.168.206.150@tcp [ 319.374296] Key type .llcrypt unregistered [ 319.375754] Key type ._llcrypt unregistered [ 329.202524] Key type ._llcrypt registered [ 329.203880] Key type .llcrypt registered [ 329.252031] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 329.654229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.697442] alg: No test for adler32 (adler32-zlib) [ 330.571516] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 330.683878] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 332.296148] Key type lgssc registered [ 332.724660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 335.984942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 335.994762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.213843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 338.887497] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 342.451892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.612986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.293614] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 347.577261] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 349.771707] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 352.496247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 352.671554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.112520] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 358.122384] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 362.920410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 373.220795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.222989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 373.226657] Lustre: Skipped 1 previous similar message [ 373.231639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 375.257262] Lustre: server umount lustre-OST0000 complete [ 378.339499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.340189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.340931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.340938] Lustre: Skipped 1 previous similar message [ 378.346894] Lustre: Skipped 1 previous similar message [ 383.457943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.459816] Lustre: Skipped 2 previous similar messages [ 388.577052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.580548] Lustre: Skipped 1 previous similar message [ 391.136114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 391.217368] Lustre: server umount lustre-MDT0000 complete [ 392.713116] LustreError: 24208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866384 with bad export cookie 10881156106550917226 [ 392.716233] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.717415] LustreError: 24208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 392.841356] Lustre: server umount lustre-MDT0001 complete [ 397.007903] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 398.381757] Key type lgssc unregistered [ 398.541629] LNet: 28197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.546969] LNetError: 28197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 399.592209] LNet: Removed LNI 192.168.206.150@tcp [ 400.046163] Key type .llcrypt unregistered [ 400.048033] Key type ._llcrypt unregistered [ 412.461660] Key type ._llcrypt registered [ 412.463266] Key type .llcrypt registered [ 412.529770] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 413.041708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 413.050513] alg: No test for adler32 (adler32-zlib) [ 413.937818] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 414.067112] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 415.672150] Key type lgssc registered [ 416.106947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 419.442943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 419.451106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.688137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.516141] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 426.136782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.312803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.102649] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 431.344201] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.667275] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 438.884715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 439.014750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.182501] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 444.050167] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.161802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 455.650682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 455.656648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.666054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.674132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.675973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.840366] Lustre: server umount lustre-OST0000 complete [ 461.794904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.801785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.804462] Lustre: Skipped 2 previous similar messages [ 466.430534] Lustre: server umount lustre-MDT0000 complete [ 466.916197] LustreError: 29585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.928739] LustreError: 29585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 468.272604] LustreError: 29566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866460 with bad export cookie 1157559253789536708 [ 468.274727] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 468.287016] LustreError: 29566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 468.447611] Lustre: server umount lustre-MDT0001 complete [ 473.231996] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 474.498296] Key type lgssc unregistered [ 474.664697] LNet: 33456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.673107] LNetError: 33456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.680500] LNet: Removed LNI 192.168.206.150@tcp [ 475.061321] Key type .llcrypt unregistered [ 475.063984] Key type ._llcrypt unregistered [ 485.640092] Key type ._llcrypt registered [ 485.641951] Key type .llcrypt registered [ 485.697687] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 486.199798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 486.206872] alg: No test for adler32 (adler32-zlib) [ 487.125057] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 487.242310] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 488.840178] Key type lgssc registered [ 489.334354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 492.871606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 492.881576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.119265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.911731] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 499.329768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.505866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 501.055602] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 504.116425] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 506.534018] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.475307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.664170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.442727] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 515.815730] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 518.897869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 526.306898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 526.312050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.330048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.379422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.381677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.385781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.396720] Lustre: Skipped 1 previous similar message [ 532.002083] Lustre: server umount lustre-OST0000 complete [ 534.504135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.507250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 534.518923] Lustre: Skipped 1 previous similar message [ 534.528777] Lustre: Skipped 1 previous similar message [ 539.618406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.623073] Lustre: Skipped 1 previous similar message [ 544.738936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.741364] Lustre: Skipped 1 previous similar message [ 548.320127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 548.397492] Lustre: server umount lustre-MDT0000 complete [ 549.858930] LustreError: 35572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.876098] LustreError: 35572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 549.951623] LustreError: 34822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866542 with bad export cookie 14272487377139139039 [ 549.952907] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 549.957859] LustreError: 34822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 550.109075] Lustre: server umount lustre-MDT0001 complete [ 554.737923] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 555.865859] Key type lgssc unregistered [ 556.008764] LNet: 38370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.013313] LNetError: 38370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.020752] LNet: Removed LNI 192.168.206.150@tcp [ 556.422405] Key type .llcrypt unregistered [ 556.423649] Key type ._llcrypt unregistered [ 566.766842] Key type ._llcrypt registered [ 566.768534] Key type .llcrypt registered [ 566.818421] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 567.201532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 567.235057] alg: No test for adler32 (adler32-zlib) [ 568.107703] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 568.207491] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 569.792700] Key type lgssc registered [ 570.287700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.847290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.863453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.119264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.043901] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 581.005813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.174047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 582.978847] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 586.150554] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.626136] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.380236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 591.537996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.735502] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 596.500169] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 600.743093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 611.300542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.300963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.307157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.316200] Lustre: Skipped 1 previous similar message [ 613.245600] Lustre: lustre-OST0000: Not available for connect from 192.168.206.50@tcp (stopping) [ 615.261485] Lustre: server umount lustre-OST0000 complete [ 628.876961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.882042] Lustre: Skipped 2 previous similar messages [ 628.882562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.887378] Lustre: Skipped 1 previous similar message [ 630.945981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.949225] Lustre: Skipped 1 previous similar message [ 635.045803] Lustre: server umount lustre-MDT0000 complete [ 636.385061] LustreError: 39756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.391973] LustreError: 39756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 636.679333] LustreError: 41596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866628 with bad export cookie 1614396659173454441 [ 636.680825] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.686151] LustreError: 41596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 642.869899] Lustre: server umount lustre-MDT0001 complete [ 646.590576] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 647.675749] Key type lgssc unregistered [ 647.807648] LNet: 43484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.811132] LNetError: 43484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.823856] LNet: Removed LNI 192.168.206.150@tcp [ 648.150165] Key type .llcrypt unregistered [ 648.151626] Key type ._llcrypt unregistered [ 657.544799] Key type ._llcrypt registered [ 657.546099] Key type .llcrypt registered [ 657.587529] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 657.896727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 657.905402] alg: No test for adler32 (adler32-zlib) [ 658.780304] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 658.881715] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 660.472135] Key type lgssc registered [ 660.860769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 663.723291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 663.730805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.908213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.450329] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 669.657767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.794866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 671.260144] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 673.834233] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 675.943599] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 678.561436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.700316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.840664] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 683.870149] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 689.968540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 699.765713] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.50@tcp (stopping) [ 700.387294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 700.387338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 700.394868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.400544] Lustre: Skipped 2 previous similar messages [ 703.466310] Lustre: server umount lustre-MDT0000 complete [ 704.883020] LustreError: 44867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.890620] LustreError: 44867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 705.195235] LustreError: 44850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866697 with bad export cookie 11990931703970628418 [ 705.199316] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.202872] LustreError: 44850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 705.505184] LustreError: 44868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.505496] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.529020] Lustre: Skipped 2 previous similar messages [ 705.532582] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 709.999608] LustreError: 45598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.007848] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.50@tcp (stopping) [ 711.364148] Lustre: server umount lustre-MDT0001 complete [ 729.652230] Lustre: server umount lustre-OST0000 complete [ 736.396802] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 737.734854] Key type lgssc unregistered [ 737.879358] LNet: 48745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.882244] LNetError: 48745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.891360] LNet: Removed LNI 192.168.206.150@tcp [ 738.256611] Key type .llcrypt unregistered [ 738.258825] Key type ._llcrypt unregistered [ 749.411945] Key type ._llcrypt registered [ 749.413296] Key type .llcrypt registered [ 749.464505] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 749.907543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.919118] alg: No test for adler32 (adler32-zlib) [ 750.807973] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 750.926861] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 752.520219] Key type lgssc registered [ 753.042747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 756.666925] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 756.679152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.192377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 777.312280] LustreError: 50103:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 803.936241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.918562] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 810.014398] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 976.241529] Lustre: server umount lustre-OST0000 complete [ 982.418290] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 983.618937] Key type lgssc unregistered [ 983.763498] LNet: 51779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.766979] LNetError: 51779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.776585] LNet: Removed LNI 192.168.206.150@tcp [ 984.142527] Key type .llcrypt unregistered [ 984.145809] Key type ._llcrypt unregistered [ 994.559993] Key type ._llcrypt registered [ 994.561411] Key type .llcrypt registered [ 994.604824] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 994.922937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 994.952275] alg: No test for adler32 (adler32-zlib) [ 995.810083] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 995.900199] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 997.488202] Key type lgssc registered [ 997.910485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1000.803540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1000.813284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.966335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.478139] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1006.878794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.043053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.688929] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1011.605407] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1013.906866] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1016.761766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.916640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.225807] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1022.008058] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1023.460954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1023.461791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.469499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1024.995840] Lustre: server umount lustre-OST0000 complete [ 1027.553371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1027.556559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.563978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.577814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.580764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.662659] Lustre: server umount lustre-MDT0000 complete [ 1033.697279] LustreError: 53184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.704962] LustreError: 53184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1034.083382] LustreError: 53147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867026 with bad export cookie 14785706745550219179 [ 1034.087242] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.089515] LustreError: 53147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1034.215647] Lustre: server umount lustre-MDT0001 complete [ 1037.960415] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1039.130562] Key type lgssc unregistered [ 1039.288678] LNet: 56689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.296203] LNetError: 56689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.304745] LNet: Removed LNI 192.168.206.150@tcp [ 1039.701306] Key type .llcrypt unregistered [ 1039.703416] Key type ._llcrypt unregistered [ 1050.348256] Key type ._llcrypt registered [ 1050.349748] Key type .llcrypt registered [ 1050.406258] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1050.779292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1050.850039] alg: No test for adler32 (adler32-zlib) [ 1051.750351] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1051.865093] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1053.464866] Key type lgssc registered [ 1053.920350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.269882] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1057.280829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.736268] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1077.857377] LustreError: 58047:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 1104.480871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.178298] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1110.026480] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.872640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.120358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1113.156742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1114.884205] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1118.177925] LustreError: 58072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.184879] LustreError: 58072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1118.263995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.439454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.068934] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1122.919627] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1125.152662] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1128.416994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.421155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.428517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.929223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.932581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.939195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.954172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.075289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1135.078336] Lustre: Skipped 2 previous similar messages [ 1140.192106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1140.193423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1140.201724] Lustre: Skipped 1 previous similar message [ 1140.261283] Lustre: server umount lustre-OST0000 complete [ 1154.654176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.660154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.649827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1159.653661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.920861] Lustre: server umount lustre-MDT0000 complete [ 1162.327176] LustreError: 58053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867154 with bad export cookie 14413410522837177469 [ 1162.330860] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.331312] LustreError: 58053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1168.500445] Lustre: server umount lustre-MDT0001 complete [ 1172.196837] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1173.322127] Key type lgssc unregistered [ 1173.467426] LNet: 61786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.470309] LNetError: 61786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.477608] LNet: Removed LNI 192.168.206.150@tcp [ 1173.834627] Key type .llcrypt unregistered [ 1173.836146] Key type ._llcrypt unregistered [ 1183.492964] Key type ._llcrypt registered [ 1183.494121] Key type .llcrypt registered [ 1183.530308] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1183.872403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1183.919686] alg: No test for adler32 (adler32-zlib) [ 1184.798306] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1184.903330] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1186.496171] Key type lgssc registered [ 1186.905884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1189.999174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1190.012293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.220312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.828271] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1196.377389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.582590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.460281] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1201.411392] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1203.520981] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1206.231293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1206.414691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.696522] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1211.678776] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1217.707367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1228.261173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.262053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.264581] LustreError: Skipped 1 previous similar message [ 1228.270436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.275857] Lustre: Skipped 1 previous similar message [ 1230.554530] Lustre: server umount lustre-OST0000 complete [ 1232.352960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1232.356380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.361065] Lustre: Skipped 1 previous similar message [ 1232.363745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.377767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.378016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.170112] Lustre: server umount lustre-MDT0000 complete [ 1238.496951] LustreError: 65047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.503547] LustreError: 65047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1239.616812] LustreError: 65669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867231 with bad export cookie 13206314054514037772 [ 1239.622080] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.624132] LustreError: 65669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1239.763392] Lustre: server umount lustre-MDT0001 complete [ 1243.980430] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1245.116780] Key type lgssc unregistered [ 1245.259524] LNet: 66722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.262502] LNetError: 66722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.272392] LNet: Removed LNI 192.168.206.150@tcp [ 1245.640841] Key type .llcrypt unregistered [ 1245.642482] Key type ._llcrypt unregistered [ 1256.885354] Key type ._llcrypt registered [ 1256.886928] Key type .llcrypt registered [ 1256.933747] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1257.387158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.398582] alg: No test for adler32 (adler32-zlib) [ 1258.285754] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1258.389640] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1259.976215] Key type lgssc registered [ 1260.456621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.079521] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:34:16 (1788867256) [ 1266.851635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1266.858603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.044981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.586338] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1272.922880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.096531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1274.565639] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1277.375629] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.742398] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.626293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.783600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.261826] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1288.361433] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1293.044914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1295.815724] Lustre: Failing over lustre-MDT0000 [ 1296.007728] Lustre: server umount lustre-MDT0000 complete [ 1298.405889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.407208] LustreError: 70887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.413057] Lustre: Skipped 1 previous similar message [ 1298.422081] LustreError: 70887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1299.398918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.462522] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.532207] Lustre: *** cfs_fail_loc=135, val=0*** [ 1299.573845] LustreError: 71487:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1299.577469] LustreError: 71487:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.150@tcp: cfg command failed: rc = -2 [ 1299.580892] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1299.580892] [ 1299.585253] LustreError: MGC192.168.206.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1299.592106] LustreError: 71475:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1299.596306] LustreError: 71475:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 1299.600130] LustreError: 71475:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1299.602757] LustreError: 71475:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1299.618545] Lustre: server umount lustre-MDT0000 complete [ 1299.620480] LustreError: 71475:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1302.189982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1302.273931] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1302.384404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1302.408621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1303.901233] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1303.916357] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1307.625050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1307.642686] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1307.663524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1309.665554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.668512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.673313] Lustre: Skipped 1 previous similar message [ 1309.676151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.737770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1312.740305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.743512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.899249] Lustre: server umount lustre-OST0000 complete [ 1317.858021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.860640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.867664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.872544] Lustre: Skipped 2 previous similar messages [ 1322.464725] Lustre: server umount lustre-MDT0000 complete [ 1322.977851] LustreError: 68396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.989979] LustreError: 68396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1323.941581] LustreError: 70238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867316 with bad export cookie 571864632789480484 [ 1323.944624] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.947741] LustreError: 70238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1324.074055] Lustre: server umount lustre-MDT0001 complete [ 1327.972378] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1329.182584] Key type lgssc unregistered [ 1329.331117] LNet: 73263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.335886] LNetError: 73263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.347374] LNet: Removed LNI 192.168.206.150@tcp [ 1329.726275] Key type .llcrypt unregistered [ 1329.731322] Key type ._llcrypt unregistered [ 1339.560124] Key type ._llcrypt registered [ 1339.561383] Key type .llcrypt registered [ 1339.602498] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1340.028828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1340.062103] alg: No test for adler32 (adler32-zlib) [ 1340.936047] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1341.040180] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1342.632199] Key type lgssc registered [ 1343.014823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.815254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.822387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.984253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.474039] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1351.675293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.828746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.343872] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1356.164898] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1358.365876] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1360.944793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1361.092977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.407689] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1366.382380] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1372.396330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1379.667990] Lustre: Failing over lustre-MDT0000 [ 1379.806431] Lustre: server umount lustre-MDT0000 complete [ 1382.259589] LustreError: 74643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.266588] LustreError: 74643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1382.880968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1382.886038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.887350] LustreError: 74666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.902630] Lustre: Skipped 1 previous similar message [ 1384.030790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.102948] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.184489] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1384.239323] LustreError: 77752:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1384.242587] LustreError: 77752:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.150@tcp: cfg command failed: rc = -9 [ 1384.246846] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1384.246846] [ 1384.251236] LustreError: MGC192.168.206.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1384.257473] LustreError: 77741:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1384.261373] LustreError: 77741:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 1384.266237] LustreError: 77741:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1384.270077] LustreError: 77741:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1384.284315] Lustre: server umount lustre-MDT0000 complete [ 1384.290241] LustreError: 77741:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1387.246295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.315436] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.372368] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.50@tcp (not set up) [ 1387.453889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.484178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1389.181698] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1392.495565] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1392.616834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1392.630275] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1392.650990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1397.733224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.733952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.745354] Lustre: Skipped 1 previous similar message [ 1397.748165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.750895] Lustre: Skipped 1 previous similar message [ 1400.028930] Lustre: server umount lustre-OST0000 complete [ 1402.851671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.853417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1402.854324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1402.858586] Lustre: Skipped 2 previous similar messages [ 1407.714979] Lustre: server umount lustre-MDT0000 complete [ 1407.969476] LustreError: 74644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.976438] LustreError: 74644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1409.266363] LustreError: 74629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867401 with bad export cookie 13924406785770689145 [ 1409.269412] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1409.274959] LustreError: 74629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1409.409813] Lustre: server umount lustre-MDT0001 complete [ 1413.366144] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1414.567888] Key type lgssc unregistered [ 1414.710569] LNet: 79526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1414.714745] LNetError: 79526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1414.726684] LNet: Removed LNI 192.168.206.150@tcp [ 1415.091472] Key type .llcrypt unregistered [ 1415.093237] Key type ._llcrypt unregistered [ 1419.645561] Key type ._llcrypt registered [ 1419.646895] Key type .llcrypt registered [ 1419.698604] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1419.825305] Key type .llcrypt unregistered [ 1419.826880] Key type ._llcrypt unregistered [ 1429.878613] Key type ._llcrypt registered [ 1429.880179] Key type .llcrypt registered [ 1429.922708] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1430.363561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1430.374462] alg: No test for adler32 (adler32-zlib) [ 1431.240384] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1431.352896] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1432.936151] Key type lgssc registered [ 1433.336083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.075052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.085470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.231938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.602260] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1441.658784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1441.791167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1443.273910] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1446.018622] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.189898] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1451.162517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.385308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.207757] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1457.329534] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1463.656240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1471.970339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.970370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1471.976585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1471.982215] Lustre: Skipped 1 previous similar message [ 1477.089800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.092374] Lustre: Skipped 1 previous similar message [ 1477.538199] Lustre: server umount lustre-OST0000 complete [ 1482.209584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.212379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.215966] Lustre: Skipped 2 previous similar messages [ 1482.221374] Lustre: Skipped 1 previous similar message [ 1485.093509] Lustre: server umount lustre-MDT0000 complete [ 1486.464302] LustreError: 81375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867478 with bad export cookie 5660733207795554210 [ 1486.468050] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1486.470586] LustreError: 81375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1486.588477] Lustre: server umount lustre-MDT0001 complete [ 1490.697201] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1491.860515] Key type lgssc unregistered [ 1492.004443] LNet: 84942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.007883] LNetError: 84942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.015400] LNet: Removed LNI 192.168.206.150@tcp [ 1492.365979] Key type .llcrypt unregistered [ 1492.368660] Key type ._llcrypt unregistered [ 1501.594313] Key type ._llcrypt registered [ 1501.595622] Key type .llcrypt registered [ 1501.640348] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1502.084755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.093838] alg: No test for adler32 (adler32-zlib) [ 1502.965895] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1503.055653] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1504.640138] Key type lgssc registered [ 1505.045364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1508.005211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1508.015149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.199106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.803167] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1514.079271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.223993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.766305] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1518.337657] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1520.202668] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1522.561288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1522.682040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.760454] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1527.558507] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1533.928692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1544.161846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1544.164529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.166319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1544.170304] Lustre: Skipped 1 previous similar message [ 1544.174677] Lustre: Skipped 1 previous similar message [ 1547.036241] Lustre: server umount lustre-OST0000 complete [ 1549.284285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.294670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.297905] Lustre: Skipped 1 previous similar message [ 1554.401833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1554.406432] Lustre: Skipped 1 previous similar message [ 1554.783406] Lustre: server umount lustre-MDT0000 complete [ 1556.452585] LustreError: 89131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867548 with bad export cookie 16323257228065483882 [ 1556.453474] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.461523] LustreError: 89131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1556.621524] Lustre: server umount lustre-MDT0001 complete [ 1561.861761] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1563.251537] Key type lgssc unregistered [ 1563.416450] LNet: 89881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.423284] LNetError: 89881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.436777] LNet: Removed LNI 192.168.206.150@tcp [ 1563.894435] Key type .llcrypt unregistered [ 1563.896239] Key type ._llcrypt unregistered [ 1575.425344] Key type ._llcrypt registered [ 1575.426852] Key type .llcrypt registered [ 1575.476164] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1575.940919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.958365] alg: No test for adler32 (adler32-zlib) [ 1576.852629] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1576.969931] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1578.568127] Key type lgssc registered [ 1579.017640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1582.181815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1582.192114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.377972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.183427] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1588.779116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.962925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1590.585272] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1593.835628] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1596.215137] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1599.168275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.341347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.847406] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1604.939465] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1608.559392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1619.938662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.939694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1619.941291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1619.943860] Lustre: Skipped 1 previous similar message [ 1625.059473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1625.062733] Lustre: Skipped 2 previous similar messages [ 1625.942675] Lustre: server umount lustre-OST0000 complete [ 1630.179520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.179576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1630.187032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.193077] Lustre: Skipped 1 previous similar message [ 1633.772219] Lustre: server umount lustre-MDT0000 complete [ 1635.297101] LustreError: 91266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.305357] LustreError: 91266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1635.422385] LustreError: 93107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867627 with bad export cookie 5441901430590735766 [ 1635.428679] LustreError: 93107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1635.429408] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.591871] Lustre: server umount lustre-MDT0001 complete [ 1640.102266] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1641.344826] Key type lgssc unregistered [ 1641.483402] LNet: 94816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.486521] LNetError: 94816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.493392] LNet: Removed LNI 192.168.206.150@tcp [ 1641.877076] Key type .llcrypt unregistered [ 1641.879051] Key type ._llcrypt unregistered [ 1652.495367] Key type ._llcrypt registered [ 1652.496854] Key type .llcrypt registered [ 1652.556429] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1652.924065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1652.960484] alg: No test for adler32 (adler32-zlib) [ 1653.851763] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1653.961196] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1655.560152] Key type lgssc registered [ 1656.028767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1659.341982] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1659.349965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1674.784380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1679.904312] LustreError: 96174:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 1706.529060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.334457] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1712.736941] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1715.441059] Lustre: server umount lustre-OST0000 complete [ 1719.083975] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:41:51 (1788867711) [ 1722.130924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.360914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.357084] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1727.462917] LustreError: 97811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.479089] LustreError: 97811:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1728.507355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.671860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.298497] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1732.941499] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1734.854779] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1737.487238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1737.646989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.997544] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1743.017811] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1746.860213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1754.082313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.090453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.097653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.156748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.158388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.163540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.181896] Lustre: Skipped 1 previous similar message [ 1762.273876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1762.282027] Lustre: Skipped 1 previous similar message [ 1767.392177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1767.397246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1767.399863] Lustre: Skipped 1 previous similar message [ 1767.469559] Lustre: server umount lustre-OST0000 complete [ 1769.953598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1769.956993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.518450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.518922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1772.531280] Lustre: Skipped 2 previous similar messages [ 1775.548821] Lustre: server umount lustre-MDT0000 complete [ 1777.578325] LustreError: 97792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867769 with bad export cookie 18145828430954000322 [ 1777.579138] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.589106] LustreError: 97792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1777.769684] Lustre: server umount lustre-MDT0001 complete [ 1782.894582] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1784.302224] Key type lgssc unregistered [ 1784.455465] LNet: 101497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.461579] LNetError: 101497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.476165] LNet: Removed LNI 192.168.206.150@tcp [ 1784.881416] Key type .llcrypt unregistered [ 1784.883521] Key type ._llcrypt unregistered [ 1804.347551] Key type ._llcrypt registered [ 1804.349147] Key type .llcrypt registered [ 1804.399480] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1804.884267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1804.910764] alg: No test for adler32 (adler32-zlib) [ 1805.801113] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1805.918794] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1807.520445] Key type lgssc registered [ 1808.114069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1811.827761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.843086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.068093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.962044] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1819.264160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1819.464669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.516960] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1825.241134] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1828.078917] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1831.457259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.648480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.431268] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1837.726590] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1837.799844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1848.289511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.293256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.302241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.313958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.315354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.319135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.090777] Lustre: server umount lustre-OST0000 complete [ 1854.433623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.441955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1854.444519] Lustre: Skipped 2 previous similar messages [ 1859.552918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.558767] Lustre: Skipped 1 previous similar message [ 1864.673774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1864.676754] Lustre: Skipped 1 previous similar message [ 1868.256664] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1868.340590] Lustre: server umount lustre-MDT0000 complete [ 1869.795110] LustreError: 103661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.803300] LustreError: 103661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1870.241648] LustreError: 105500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867862 with bad export cookie 14268683110667996985 [ 1870.242179] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.249087] LustreError: 105500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1870.382115] Lustre: server umount lustre-MDT0001 complete [ 1875.207090] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 1876.676578] Key type lgssc unregistered [ 1876.870871] LNet: 107188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.877693] LNetError: 107188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1876.893500] LNet: Removed LNI 192.168.206.150@tcp [ 1877.349657] Key type .llcrypt unregistered [ 1877.351503] Key type ._llcrypt unregistered [ 1886.586501] Key type ._llcrypt registered [ 1886.589037] Key type .llcrypt registered [ 1886.638258] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1887.038444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1887.049079] alg: No test for adler32 (adler32-zlib) [ 1887.962563] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1888.079264] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 1889.696342] Key type lgssc registered [ 1890.280251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.076337] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.091840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1909.603850] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1914.720673] LustreError: 108418:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 1941.344173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.183422] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1947.662886] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1950.970581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.121469] LustreError: MGC192.168.206.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1951.129451] LustreError: 109444:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1951.133850] LustreError: 109444:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 1951.138220] LustreError: 109444:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 1951.177670] Lustre: server umount lustre-MDT0000 complete [ 1951.180198] LustreError: 109444:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1965.610417] Lustre: server umount lustre-OST0000 complete [ 1970.167815] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 1974.481327] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 1991.819159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.941964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1991.962326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1992.016677] Lustre: lustre-MDT0000: new disk, initializing [ 1992.057919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.067297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1993.940119] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 1998.863622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.916093] Lustre: 112230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1998.949751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1998.951865] Lustre: Skipped 1 previous similar message [ 1998.993854] Lustre: lustre-MDT0001: new disk, initializing [ 1999.035699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.057734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1999.063238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2000.822662] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2005.405532] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2007.864796] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.766879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2010.895821] Lustre: lustre-OST0000: new disk, initializing [ 2010.898231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2010.902318] Lustre: 114198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2010.943961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.396569] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2016.257163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2016.262108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2016.310443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2017.580834] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2019.385414] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2019.479596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2021.233072] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2021.321686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2026.467366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.468071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2026.474281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2026.478410] LustreError: Skipped 1 previous similar message [ 2028.260527] Lustre: server umount lustre-OST0000 complete [ 2030.049269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2030.053712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.059017] Lustre: Skipped 1 previous similar message [ 2030.062421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.066749] Lustre: Skipped 1 previous similar message [ 2031.585475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.590430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.705141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.709350] Lustre: Skipped 2 previous similar messages [ 2041.826139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2041.829436] Lustre: Skipped 1 previous similar message [ 2044.384315] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2044.455377] Lustre: server umount lustre-MDT0000 complete [ 2046.203471] LustreError: 112221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868038 with bad export cookie 3408064477700992667 [ 2046.207085] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.211646] LustreError: 112221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.345667] Lustre: server umount lustre-MDT0001 complete [ 2049.815718] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:47:22 (1788868042) [ 2051.168801] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2057.848483] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 2061.755504] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 2077.372710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.469830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2077.484242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2077.523150] Lustre: lustre-MDT0000: new disk, initializing [ 2077.556705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.565877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2079.417993] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2084.303246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.356735] Lustre: 119090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2084.404770] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2084.407930] Lustre: Skipped 1 previous similar message [ 2084.446612] Lustre: lustre-MDT0001: new disk, initializing [ 2084.501466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2084.508287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2086.105500] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2090.478430] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2092.872226] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2095.881090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2096.004709] Lustre: lustre-OST0000: new disk, initializing [ 2096.007134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2096.010389] Lustre: 121053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2096.055145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.057658] Lustre: Skipped 1 previous similar message [ 2098.572671] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2103.069547] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2104.819497] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2104.829311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2104.834491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2104.881186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2105.922887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2107.505447] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2107.606751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2109.924253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2109.924505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.933133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2109.936064] Lustre: Skipped 1 previous similar message [ 2114.552824] Lustre: server umount lustre-OST0000 complete [ 2120.162392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.167889] Lustre: Skipped 2 previous similar messages [ 2130.400635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2130.402574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.409990] Lustre: Skipped 5 previous similar messages [ 2130.491113] Lustre: server umount lustre-MDT0000 complete [ 2132.113850] LustreError: 119081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868124 with bad export cookie 3408064477700993535 [ 2132.115715] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.122323] LustreError: 119081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2132.281875] Lustre: server umount lustre-MDT0001 complete [ 2136.265155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.497188] LustreError: 123275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.508649] LustreError: 123275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2136.542444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.408846] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2141.666217] LustreError: 123276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.273798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.298563] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2147.682659] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2150.149735] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2153.031648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2155.925315] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2159.295274] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2172.217302] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2176.992965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2176.996259] LustreError: Skipped 1 previous similar message [ 2176.999018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.006184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.742863] Lustre: server umount lustre-OST0000 complete [ 2183.559488] Lustre: server umount lustre-MDT0000 complete [ 2185.577079] LustreError: 123257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868177 with bad export cookie 3408064477700994403 [ 2185.581313] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.586349] LustreError: 123257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2187.764115] Lustre: server umount lustre-MDT0001 complete [ 2191.873482] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 2193.053265] Key type lgssc unregistered [ 2193.194823] LNet: 126919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.199376] LNetError: 126919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.211346] LNet: Removed LNI 192.168.206.150@tcp [ 2193.552638] Key type .llcrypt unregistered [ 2193.554079] Key type ._llcrypt unregistered [ 2201.075081] Key type ._llcrypt registered [ 2201.076822] Key type .llcrypt registered [ 2201.126164] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 2209.333209] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 2209.886616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.900563] alg: No test for adler32 (adler32-zlib) [ 2210.805683] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2210.906937] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 2212.512127] Key type lgssc registered [ 2213.205328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2229.336700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2229.348874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2230.455554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2230.472411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2230.515059] Lustre: lustre-MDT0000: new disk, initializing [ 2230.553373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.562531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2232.364850] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2237.620265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.677853] Lustre: 130042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2237.714519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2237.718335] Lustre: Skipped 1 previous similar message [ 2237.758470] Lustre: lustre-MDT0001: new disk, initializing [ 2237.796074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.815696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2237.822044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2239.542513] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2244.147409] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.532109] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.518772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.639337] Lustre: lustre-OST0000: new disk, initializing [ 2249.641807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2249.645736] Lustre: 132010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2249.683835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.199080] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2256.737428] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2256.910314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2256.914608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2256.970079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2258.588977] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2258.682895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2260.479433] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2260.560982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2261.989571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2261.992182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2262.000650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.002104] LustreError: Skipped 1 previous similar message [ 2262.008824] Lustre: Skipped 1 previous similar message [ 2263.464595] Lustre: server umount lustre-OST0000 complete [ 2267.107443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.113238] Lustre: Skipped 1 previous similar message [ 2267.118835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.125347] Lustre: Skipped 1 previous similar message [ 2272.225935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.228912] Lustre: Skipped 1 previous similar message [ 2277.345693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2277.350874] Lustre: Skipped 1 previous similar message [ 2279.392660] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2279.462741] Lustre: server umount lustre-MDT0000 complete [ 2281.012981] LustreError: 130034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868273 with bad export cookie 46199511068692730 [ 2281.014972] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.016651] LustreError: 130034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2281.148187] Lustre: server umount lustre-MDT0001 complete [ 2285.014172] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:51:17 (1788868277) [ 2288.286127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.552676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2290.487910] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2293.730750] LustreError: 134421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.747381] LustreError: 134421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2294.907028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.148855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2297.188658] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2300.911444] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2304.064643] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2305.505517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2305.508711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.512045] Lustre: Skipped 1 previous similar message [ 2305.516340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.277646] Lustre: server umount lustre-MDT0000 complete [ 2311.649905] LustreError: 135507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.661660] LustreError: 135507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2313.043389] LustreError: 134401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868305 with bad export cookie 46199511068693598 [ 2313.047568] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.050932] LustreError: 134401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2313.219725] Lustre: server umount lustre-MDT0001 complete [ 2317.433143] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:51:49 (1788868309) [ 2320.537787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.032724] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2341.152440] LustreError: 136831:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 2366.752605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.859148] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2373.378667] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2374.346642] Lustre: server umount lustre-OST0000 complete [ 2378.056321] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:52:50 (1788868370) [ 2380.720023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2380.929465] LustreError: 138273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.506485] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2386.002264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2386.174505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2386.177464] Lustre: Skipped 1 previous similar message [ 2387.883181] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2391.153731] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2393.580795] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2396.722089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.565341] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2402.866654] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2416.610177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.612171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.616982] Lustre: Skipped 3 previous similar messages [ 2416.621196] Lustre: Skipped 3 previous similar messages [ 2422.649918] Lustre: server umount lustre-MDT0000 complete [ 2424.562857] LustreError: 138254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868416 with bad export cookie 46199511068694249 [ 2424.564344] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.569650] LustreError: 138254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2424.729573] Lustre: server umount lustre-MDT0001 complete [ 2436.003472] Lustre: server umount lustre-OST0000 complete [ 2439.574767] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:53:51 (1788868431) [ 2442.332977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.553752] LustreError: 141942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.585167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.588306] Lustre: Skipped 1 previous similar message [ 2444.267177] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2447.842943] LustreError: 141943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.095710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.205840] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2453.764493] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2456.284133] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2459.459774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2462.349365] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2465.547694] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.794215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2467.613455] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2467.726463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2469.614167] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2473.828537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 2477.025484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.031831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.042548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.045078] Lustre: Skipped 3 previous similar messages [ 2480.791825] Lustre: server umount lustre-OST0000 complete [ 2488.680643] Lustre: server umount lustre-MDT0000 complete [ 2490.221160] LustreError: 141924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868482 with bad export cookie 46199511068695299 [ 2490.222967] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.228339] LustreError: 141924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2490.369869] Lustre: server umount lustre-MDT0001 complete [ 2494.187903] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:54:46 (1788868486) [ 2497.514407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.992585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2518.112426] LustreError: 146123:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 2543.712331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.717198] Lustre: Skipped 2 previous similar messages [ 2546.159440] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2549.069973] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2551.765287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.948238] LustreError: 146148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.017284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2553.779097] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2557.594245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.606716] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2562.794089] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2565.286872] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.305381] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2567.405833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2569.329236] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2569.424458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2573.281688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2573.286613] LustreError: Skipped 1 previous similar message [ 2573.289292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.296368] Lustre: Skipped 3 previous similar messages [ 2573.299621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.302523] Lustre: Skipped 6 previous similar messages [ 2576.411920] Lustre: server umount lustre-MDT0000 complete [ 2577.864968] LustreError: 149269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868570 with bad export cookie 46199511068696314 [ 2577.869866] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.871319] LustreError: 149269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2578.003522] Lustre: server umount lustre-MDT0001 complete [ 2589.868497] Lustre: server umount lustre-OST0000 complete [ 2593.800758] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:56:26 (1788868586) [ 2597.089612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.577167] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2617.696441] LustreError: 150314:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 2645.950940] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2649.067444] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2651.945048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2652.131866] LustreError: 150339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.142055] LustreError: 150339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2652.197987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2653.814154] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2657.172979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.991403] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2661.954463] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2664.132740] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2666.683062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.770217] Lustre: lustre-OST0001: new disk, initializing [ 2666.773421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2666.777212] Lustre: 153184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2667.903261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2667.909161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2667.953803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2669.269505] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2673.767152] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2675.534335] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2675.618306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2677.239684] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2677.330227] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2678.244603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.248095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.250538] Lustre: Skipped 3 previous similar messages [ 2678.253098] Lustre: Skipped 3 previous similar messages [ 2684.169811] Lustre: server umount lustre-OST0000 complete [ 2688.484576] LustreError: 150330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.486192] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.492895] LustreError: 150330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2688.501993] LustreError: Skipped 1 previous similar message [ 2700.256154] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2700.343806] Lustre: server umount lustre-OST0001 complete [ 2703.328909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2716.128130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2716.206024] Lustre: server umount lustre-MDT0000 complete [ 2717.982591] LustreError: 150321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868710 with bad export cookie 46199511068697266 [ 2717.984941] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2717.988120] LustreError: 150321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2718.138884] Lustre: server umount lustre-MDT0001 complete [ 2727.923531] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:58:40 (1788868720) [ 2728.640396] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2729.258909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2729.916930] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:58:42 (1788868722) [ 2732.674830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.801283] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2732.815255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2732.912531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.915497] Lustre: Skipped 6 previous similar messages [ 2734.664330] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2738.192976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.253222] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2738.267949] Lustre: 156738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2740.138800] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2743.421291] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2745.833249] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2748.702253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2748.809905] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2751.281609] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2754.556963] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2756.350605] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2756.644080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2757.474215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2759.239341] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2759.341893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2761.698750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2761.700518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2761.706777] LustreError: Skipped 1 previous similar message [ 2761.712520] Lustre: Skipped 5 previous similar messages [ 2766.297621] Lustre: server umount lustre-OST0000 complete [ 2780.885200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2782.057971] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.066215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2782.066336] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2782.075945] Lustre: Skipped 1 previous similar message [ 2783.696782] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2786.766181] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2788.848265] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2788.957454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2790.864338] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2790.966377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2793.413281] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c2586f4e000.ost_server_uuid 50 [ 2794.115678] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c2586f4e000.ost_server_uuid in FULL state after 0 sec [ 2794.197046] LustreError: 159208:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.50@tcp) returned error from blocking AST (req@00000000981adcf1 x1875764265028608 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000cb0e425a/0xa42235e60bebfe lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.50@tcp remote: 0xae6b24b270aa4c2d expref: 5 pid: 156746 timeout: 2894 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2794.215230] LustreError: lustre-MDT0000: A client on nid 192.168.206.50@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2794.219143] LustreError: 156735:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.50@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000cb0e425a/0xa42235e60bebfe lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.50@tcp remote: 0xae6b24b270aa4c2d expref: 6 pid: 156746 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2797.545032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.095802] Lustre: server umount lustre-OST0000 complete [ 2807.778063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.781133] Lustre: Skipped 21 previous similar messages [ 2818.016162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2818.092819] Lustre: server umount lustre-MDT0000 complete [ 2819.743527] LustreError: 156732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868811 with bad export cookie 46199511068698449 [ 2819.747056] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.749108] LustreError: 156732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2833.888408] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2834.003749] Lustre: server umount lustre-MDT0001 complete [ 2838.539917] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 2839.908776] Key type lgssc unregistered [ 2840.065719] LNet: 162848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.070564] LNetError: 162848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.081826] LNet: Removed LNI 192.168.206.150@tcp [ 2840.485873] Key type .llcrypt unregistered [ 2840.487875] Key type ._llcrypt unregistered [ 2851.094947] Key type ._llcrypt registered [ 2851.096798] Key type .llcrypt registered [ 2851.144256] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 2851.593170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2851.600570] alg: No test for adler32 (adler32-zlib) [ 2852.468339] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2852.571509] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 2854.160224] Key type lgssc registered [ 2854.618844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2857.896452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2857.911758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.108847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.894544] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2864.535385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.711744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2866.425372] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2869.796799] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2872.025520] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2875.131470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.293780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2877.995431] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2881.420994] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2884.523089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2895.263830] Lustre: Failing over lustre-MDT0000 [ 2895.329304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2895.333137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.341665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.451600] Lustre: server umount lustre-MDT0000 complete [ 2899.940513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.940670] LustreError: 164237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.961089] LustreError: 164237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2905.058046] LustreError: 166735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.065336] LustreError: 166735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2906.618287] LustreError: 166848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.50@tcp arrived at 1788868898 with bad export cookie 16959351584416167877 [ 2906.628365] LustreError: 166848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2910.051326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.120388] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.177859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2910.180636] Lustre: Skipped 2 previous similar messages [ 2910.238373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.264747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2912.107986] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2913.191662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2913.195089] Lustre: lustre-MDT0000: Denying connection for new client 3d40d683-e2f2-47b1-b94b-9a299934f0b3 (at 192.168.206.50@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2915.307902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2920.418657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.418814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.420539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.433877] Lustre: Skipped 2 previous similar messages [ 2925.538117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.540579] Lustre: Skipped 2 previous similar messages [ 2926.320610] Lustre: server umount lustre-OST0000 complete [ 2927.938795] LustreError: 167996:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2927.942429] Lustre: 167346:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2927.947328] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2927.959811] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2927.965154] LustreError: 167346:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000598b5f8c x1875764937003264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2927.975970] LustreError: 167346:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2927.982063] LustreError: 167346:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2927.986576] Lustre: lustre-MDT0000: Recovery over after 0:14, of 2 clients 0 recovered and 2 were evicted. [ 2928.147196] Lustre: server umount lustre-MDT0000 complete [ 2930.021301] LustreError: 166848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868922 with bad export cookie 16959351584416168241 [ 2930.027190] LustreError: 166848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2930.032103] LustreError: 164237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.032385] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.265341] Lustre: server umount lustre-MDT0001 complete [ 2941.034203] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 2942.502204] Key type lgssc unregistered [ 2942.675544] LNet: 168746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.680276] LNetError: 168746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.693505] LNet: Removed LNI 192.168.206.150@tcp [ 2943.115178] Key type .llcrypt unregistered [ 2943.117264] Key type ._llcrypt unregistered [ 2954.008141] Key type ._llcrypt registered [ 2954.009572] Key type .llcrypt registered [ 2954.061612] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 2954.480927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2954.511072] alg: No test for adler32 (adler32-zlib) [ 2955.411825] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2955.542308] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 2957.152192] Key type lgssc registered [ 2957.622242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2961.118293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2961.130459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.371589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.404853] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2968.090646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.274885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.885966] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2972.777716] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2975.188279] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2978.082537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2978.228760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2980.501524] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 2983.501202] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2987.431599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2992.610042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2992.611341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.614883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.618704] Lustre: Skipped 1 previous similar message [ 2992.624012] Lustre: Skipped 1 previous similar message [ 2996.224408] Lustre: server umount lustre-OST0000 complete [ 2999.265338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2999.269723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.277235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.853577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.855399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.151767] Lustre: server umount lustre-MDT0000 complete [ 3005.817277] LustreError: 170113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868998 with bad export cookie 8892870883159833793 [ 3005.820742] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.825538] LustreError: 170113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3005.991733] Lustre: server umount lustre-MDT0001 complete [ 3010.412368] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3011.679357] Key type lgssc unregistered [ 3011.825520] LNet: 173659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.829668] LNetError: 173659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.842443] LNet: Removed LNI 192.168.206.150@tcp [ 3012.248485] Key type .llcrypt unregistered [ 3012.249940] Key type ._llcrypt unregistered [ 3026.402483] Key type ._llcrypt registered [ 3026.404416] Key type .llcrypt registered [ 3026.450692] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3026.970733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3026.978447] alg: No test for adler32 (adler32-zlib) [ 3027.876819] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3028.001901] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3029.616126] Key type lgssc registered [ 3030.116460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3033.584275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3033.593293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.800461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.519638] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3040.163123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.331762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.029538] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3044.905321] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3046.943434] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3049.843241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3050.019050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.320196] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3055.046827] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3061.283861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3071.461581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.463063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3071.464065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3071.465239] Lustre: Skipped 1 previous similar message [ 3073.388173] Lustre: server umount lustre-OST0000 complete [ 3076.067579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.071946] LustreError: Skipped 1 previous similar message [ 3076.074613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.081213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.083913] Lustre: Skipped 1 previous similar message [ 3081.697547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.700108] Lustre: Skipped 2 previous similar messages [ 3086.817423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.821944] Lustre: Skipped 1 previous similar message [ 3089.376106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3089.448812] Lustre: server umount lustre-MDT0000 complete [ 3090.914315] LustreError: 175333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.923716] LustreError: 175333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3091.102564] LustreError: 175314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869083 with bad export cookie 5032754370755946774 [ 3091.105035] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.108465] LustreError: 175314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3091.255644] Lustre: server umount lustre-MDT0001 complete [ 3095.762701] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3096.989666] Key type lgssc unregistered [ 3097.141662] LNet: 178883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.145217] LNetError: 178883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.159462] LNet: Removed LNI 192.168.206.150@tcp [ 3097.489871] Key type .llcrypt unregistered [ 3097.491382] Key type ._llcrypt unregistered [ 3107.058712] Key type ._llcrypt registered [ 3107.060256] Key type .llcrypt registered [ 3107.101030] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3107.475649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3107.497024] alg: No test for adler32 (adler32-zlib) [ 3108.409964] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3108.551552] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3110.176757] Key type lgssc registered [ 3110.615300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3115.153362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3115.167731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.346795] Lustre: *** cfs_fail_loc=135, val=0*** [ 3116.389269] LustreError: 180378:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 3116.391980] LustreError: 180378:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.150@tcp: cfg command failed: rc = -2 [ 3116.394937] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3116.394937] [ 3116.398120] LustreError: MGC192.168.206.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3116.405981] LustreError: 180339:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3116.409094] LustreError: 180339:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3116.411848] LustreError: 180339:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3116.414969] LustreError: 180339:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3116.453399] Lustre: server umount lustre-MDT0000 complete [ 3116.455436] LustreError: 180339:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3119.529313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.748396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.624304] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3125.526589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3125.743792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.424943] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3130.880728] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3133.537970] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3135.586033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3135.596691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.600692] Lustre: Skipped 1 previous similar message [ 3140.716184] Lustre: server umount lustre-MDT0000 complete [ 3141.092201] LustreError: 181861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.101411] LustreError: 181861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3142.344407] LustreError: 180728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869134 with bad export cookie 16912383720227480447 [ 3142.347617] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3142.351811] LustreError: 180728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3142.482932] Lustre: server umount lustre-MDT0001 complete [ 3146.867762] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3148.099267] Key type lgssc unregistered [ 3148.237624] LNet: 183066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.241056] LNetError: 183066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3148.247556] LNet: Removed LNI 192.168.206.150@tcp [ 3148.624830] Key type .llcrypt unregistered [ 3148.627450] Key type ._llcrypt unregistered [ 3156.793920] Key type ._llcrypt registered [ 3156.795628] Key type .llcrypt registered [ 3156.854849] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3156.983466] Key type .llcrypt unregistered [ 3156.985432] Key type ._llcrypt unregistered [ 3164.833747] Key type ._llcrypt registered [ 3164.835185] Key type .llcrypt registered [ 3164.888527] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3165.261662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3165.271926] alg: No test for adler32 (adler32-zlib) [ 3166.156585] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3166.267065] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3167.856229] Key type lgssc registered [ 3168.344537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3171.999618] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3172.019423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3187.488361] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3192.608513] LustreError: 185167:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.150@tcp: failed processing log, type 4: rc = -110 [ 3219.232337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3222.072745] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3225.358178] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3228.360479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.610046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.640496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3230.861174] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3233.763655] LustreError: 185192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.774766] LustreError: 185192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3234.794491] LustreError: 186203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.804221] LustreError: 186203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3235.040505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.286356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3237.810091] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3241.685421] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.641307] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3246.112830] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3254.245745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.249405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.256652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3255.777992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3255.780040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3255.782337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3255.796140] Lustre: Skipped 1 previous similar message [ 3259.503437] Lustre: server umount lustre-OST0000 complete [ 3266.018964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3266.019210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.020289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3266.033469] Lustre: Skipped 1 previous similar message [ 3267.495762] Lustre: server umount lustre-MDT0000 complete [ 3269.332285] LustreError: 185175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869261 with bad export cookie 8411911263984493348 [ 3269.336659] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3269.337873] LustreError: 185175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3269.504690] Lustre: server umount lustre-MDT0001 complete [ 3274.114498] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3275.489685] Key type lgssc unregistered [ 3275.644730] LNet: 189130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.648892] LNetError: 189130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.664864] LNet: Removed LNI 192.168.206.150@tcp [ 3276.052708] Key type .llcrypt unregistered [ 3276.057056] Key type ._llcrypt unregistered [ 3287.606129] Key type ._llcrypt registered [ 3287.607428] Key type .llcrypt registered [ 3287.657932] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3288.060266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3288.071628] alg: No test for adler32 (adler32-zlib) [ 3288.972619] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3289.094099] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3290.704294] Key type lgssc registered [ 3291.298829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3295.059419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3295.068027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.283850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.937576] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3301.535569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.716474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.383194] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3306.354748] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.677171] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3311.616747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3311.810228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.075466] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3317.325089] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3321.063432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3324.861084] Lustre: Failing over lustre-MDT0000 [ 3324.972901] Lustre: server umount lustre-MDT0000 complete [ 3325.921927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3325.924787] LustreError: 192975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.930096] Lustre: Skipped 1 previous similar message [ 3325.941973] LustreError: 192975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3328.892040] LustreError: 190512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.042140] LustreError: 190516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.055638] LustreError: 190516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3333.998362] LustreError: 190512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.118676] LustreError: 190511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.126761] LustreError: 190511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3339.134143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3339.208638] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.356449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.393929] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3341.282038] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3343.355478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3344.242464] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3344.357684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3344.366021] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3344.384160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3350.043431] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3361.761104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3361.764589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.770799] Lustre: Skipped 1 previous similar message [ 3361.777150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.810649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3363.814510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.820798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3364.838208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3365.980447] Lustre: server umount lustre-OST0000 complete [ 3369.955088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.956110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3369.956335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3369.956340] Lustre: Skipped 1 previous similar message [ 3369.970366] Lustre: Skipped 1 previous similar message [ 3373.701989] Lustre: server umount lustre-MDT0000 complete [ 3375.074041] LustreError: 191753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.083637] LustreError: 191753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3375.414496] LustreError: 191247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869367 with bad export cookie 2424180520299340097 [ 3375.417390] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3375.420991] LustreError: 191247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3375.621198] Lustre: server umount lustre-MDT0001 complete [ 3380.267801] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3381.481747] Key type lgssc unregistered [ 3381.641172] LNet: 195440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3381.645689] LNetError: 195440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3381.655551] LNet: Removed LNI 192.168.206.150@tcp [ 3382.024900] Key type .llcrypt unregistered [ 3382.026580] Key type ._llcrypt unregistered [ 3393.277518] Key type ._llcrypt registered [ 3393.279123] Key type .llcrypt registered [ 3393.336800] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3393.675182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3393.701974] alg: No test for adler32 (adler32-zlib) [ 3394.603608] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3394.715613] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3396.312216] Key type lgssc registered [ 3396.803647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3400.439474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3400.452251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.671189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.457802] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3407.343060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.510488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3409.356087] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3412.745405] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3415.650137] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3419.290933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.485969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.394590] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3425.720955] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3426.669222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3428.579300] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3435.380238] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3445.640710] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3452.387430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.389770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.391336] Lustre: Skipped 1 previous similar message [ 3452.396284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3457.507089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.510186] Lustre: Skipped 2 previous similar messages [ 3458.466797] Lustre: server umount lustre-OST0000 complete [ 3462.626031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.627434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.636762] Lustre: Skipped 1 previous similar message [ 3467.747322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.750915] Lustre: Skipped 2 previous similar messages [ 3472.866703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.872804] Lustre: Skipped 1 previous similar message [ 3474.400118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3474.500334] Lustre: server umount lustre-MDT0000 complete [ 3476.174715] LustreError: 196805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869468 with bad export cookie 17495120556735132870 [ 3476.178589] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3476.182418] LustreError: 196805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3476.322817] Lustre: server umount lustre-MDT0001 complete [ 3480.678547] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3481.884666] Key type lgssc unregistered [ 3482.036579] LNet: 200675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3482.040270] LNetError: 200675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3482.051564] LNet: Removed LNI 192.168.206.150@tcp [ 3482.397870] Key type .llcrypt unregistered [ 3482.399062] Key type ._llcrypt unregistered [ 3491.157847] Key type ._llcrypt registered [ 3491.159528] Key type .llcrypt registered [ 3491.214244] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3491.606416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3491.617953] alg: No test for adler32 (adler32-zlib) [ 3492.549708] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3492.682666] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3494.280145] Key type lgssc registered [ 3494.787034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.278082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3498.290060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.491764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3501.036243] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3504.303985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.484996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3506.270604] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3509.465654] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3511.909980] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.174078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.371670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.244652] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3521.877076] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3524.589545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3530.228881] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3541.380620] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3550.389509] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3557.919651] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3570.658244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3570.660572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3570.661403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3570.666052] Lustre: Skipped 1 previous similar message [ 3573.918610] Lustre: server umount lustre-OST0000 complete [ 3575.779787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.780832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.790062] Lustre: Skipped 1 previous similar message [ 3575.794263] Lustre: Skipped 1 previous similar message [ 3580.897422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3580.900950] Lustre: Skipped 2 previous similar messages [ 3581.800179] Lustre: server umount lustre-MDT0000 complete [ 3583.404281] LustreError: 201912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869575 with bad export cookie 16364557567266903898 [ 3583.407524] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3583.409995] LustreError: 201912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3583.555242] Lustre: server umount lustre-MDT0001 complete [ 3588.190906] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3589.474760] Key type lgssc unregistered [ 3589.628916] LNet: 207429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3589.632553] LNetError: 207429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3589.640827] LNet: Removed LNI 192.168.206.150@tcp [ 3590.071870] Key type .llcrypt unregistered [ 3590.073925] Key type ._llcrypt unregistered [ 3601.918948] Key type ._llcrypt registered [ 3601.920365] Key type .llcrypt registered [ 3601.973593] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3602.315511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.323114] alg: No test for adler32 (adler32-zlib) [ 3603.213715] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3603.329800] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3604.930298] Key type lgssc registered [ 3605.547474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3609.868197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3609.882934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.137712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.102653] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3616.936799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3617.148222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3618.889284] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3622.132216] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3624.705996] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.906101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.107436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3630.761333] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3634.006870] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.326924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3646.945655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.954066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.964707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.968949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3647.975310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.976489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3647.983687] Lustre: Skipped 1 previous similar message [ 3651.698247] Lustre: server umount lustre-OST0000 complete [ 3658.210645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.214643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3658.216415] Lustre: Skipped 1 previous similar message [ 3658.216811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3658.225620] Lustre: Skipped 1 previous similar message [ 3659.695446] Lustre: server umount lustre-MDT0000 complete [ 3661.266268] LustreError: 208796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869653 with bad export cookie 6432202403800127511 [ 3661.269555] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.273056] LustreError: 208796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3661.402816] Lustre: server umount lustre-MDT0001 complete [ 3665.731764] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3666.997554] Key type lgssc unregistered [ 3667.137529] LNet: 212493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3667.142433] LNetError: 212493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3667.152544] LNet: Removed LNI 192.168.206.150@tcp [ 3667.542620] Key type .llcrypt unregistered [ 3667.544148] Key type ._llcrypt unregistered [ 3676.130074] Key type ._llcrypt registered [ 3676.131719] Key type .llcrypt registered [ 3676.184268] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3676.721319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3676.738392] alg: No test for adler32 (adler32-zlib) [ 3677.613543] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3677.723409] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3679.320737] Key type lgssc registered [ 3679.839779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3683.296466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3683.306745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.511741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.211978] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3689.832637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3690.009273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3691.804439] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3694.841934] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3697.385441] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3700.271499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3700.442324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.955658] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3706.694652] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3709.677834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3720.166082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.166444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3720.172535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3720.174659] LustreError: Skipped 1 previous similar message [ 3720.176902] Lustre: Skipped 1 previous similar message [ 3724.576310] Lustre: server umount lustre-OST0000 complete [ 3730.401764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.404520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3730.407156] Lustre: Skipped 2 previous similar messages [ 3732.459517] Lustre: server umount lustre-MDT0000 complete [ 3734.136583] LustreError: 213713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869726 with bad export cookie 11181639542421980509 [ 3734.139399] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3734.147955] LustreError: 213713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3734.293038] Lustre: server umount lustre-MDT0001 complete [ 3738.874325] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3740.165277] Key type lgssc unregistered [ 3740.314698] LNet: 217409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3740.319472] LNetError: 217409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3740.326517] LNet: Removed LNI 192.168.206.150@tcp [ 3740.706811] Key type .llcrypt unregistered [ 3740.708233] Key type ._llcrypt unregistered [ 3749.487126] Key type ._llcrypt registered [ 3749.489348] Key type .llcrypt registered [ 3749.534601] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3749.928200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3749.956132] alg: No test for adler32 (adler32-zlib) [ 3750.839317] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3750.965902] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3752.568147] Key type lgssc registered [ 3753.031741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3756.303505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3756.312089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.510826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3759.189485] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3763.064743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.251655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3765.119495] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3768.361937] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3770.762562] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3773.689111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3773.857085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.295295] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3779.367942] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3782.057334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3788.930166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3788.970615] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3789.069857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3791.502488] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3794.725651] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3796.452179] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3807.304827] Lustre: Permanently deactivating lustre-OST0001 [ 3807.323519] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3815.427398] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3821.903640] Lustre: Permanently reactivating lustre-OST0001 [ 3821.910973] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3821.914103] Lustre: Skipped 2 previous similar messages [ 3824.445255] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.451255] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3824.455191] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3824.460152] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3824.474290] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3829.729892] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.730038] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.733236] LustreError: Skipped 1 previous similar message [ 3829.738878] Lustre: Skipped 1 previous similar message [ 3829.742199] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3829.746467] Lustre: Skipped 1 previous similar message [ 3831.783350] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3831.786032] Lustre: Skipped 1 previous similar message [ 3835.397911] Lustre: server umount lustre-OST0001 complete [ 3836.899528] LustreError: 220466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3836.906763] LustreError: 220466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3838.432952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3838.441812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.451105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3839.456899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.020677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.021475] LustreError: 222138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3842.023990] Lustre: Skipped 2 previous similar messages [ 3842.031892] LustreError: 222138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3843.163345] Lustre: server umount lustre-OST0000 complete [ 3845.089421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3845.093365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.098162] Lustre: Skipped 1 previous similar message [ 3847.137864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3847.143865] Lustre: Skipped 2 previous similar messages [ 3850.819252] Lustre: server umount lustre-MDT0000 complete [ 3852.258526] LustreError: 219862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3852.272074] LustreError: 219862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3852.621024] LustreError: 218629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869844 with bad export cookie 2507137412550573344 [ 3852.624302] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.626840] LustreError: 218629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3852.772201] Lustre: server umount lustre-MDT0001 complete [ 3857.341707] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 3858.530954] Key type lgssc unregistered [ 3858.682730] LNet: 223742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.687541] LNetError: 223742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.695377] LNet: Removed LNI 192.168.206.150@tcp [ 3859.055582] Key type .llcrypt unregistered [ 3859.057078] Key type ._llcrypt unregistered [ 3875.939405] Key type ._llcrypt registered [ 3875.940715] Key type .llcrypt registered [ 3875.989322] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 3876.374400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3876.451028] alg: No test for adler32 (adler32-zlib) [ 3877.345054] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3877.457692] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 3879.056157] Key type lgssc registered [ 3879.551354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3882.811790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3882.822981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.945574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3883.957307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3884.027768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.727170] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3889.318201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.379711] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3889.400747] Lustre: 225919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3889.529324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3891.240068] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3894.351224] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.910364] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3899.907921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3900.015666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3900.107076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.758868] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 3906.215565] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3909.167786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3913.120510] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3915.711879] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3922.445990] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3930.254561] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3947.802982] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3947.807393] Lustre: Skipped 1 previous similar message [ 3966.925749] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3966.929287] Lustre: Skipped 2 previous similar messages [ 4001.061605] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4001.065130] Lustre: Skipped 3 previous similar messages [ 4069.928301] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 4069.932374] Lustre: Skipped 8 previous similar messages [ 4072.929831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.931643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.933107] LustreError: Skipped 1 previous similar message [ 4072.939064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.943336] Lustre: Skipped 1 previous similar message [ 4077.852849] Lustre: server umount lustre-OST0000 complete [ 4083.170153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.170901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.174611] Lustre: Skipped 2 previous similar messages [ 4085.477720] Lustre: server umount lustre-MDT0000 complete [ 4086.926677] LustreError: 225909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870079 with bad export cookie 4927473495331888891 [ 4086.930243] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4086.932634] LustreError: 225909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4087.059496] Lustre: server umount lustre-MDT0001 complete [ 4091.066559] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 4092.173494] Key type lgssc unregistered [ 4092.304740] LNet: 230717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4092.308735] LNetError: 230717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.317526] LNet: Removed LNI 192.168.206.150@tcp [ 4092.674494] Key type .llcrypt unregistered [ 4092.676074] Key type ._llcrypt unregistered [ 4103.215346] Key type ._llcrypt registered [ 4103.216812] Key type .llcrypt registered [ 4103.273843] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 4103.822256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.834340] alg: No test for adler32 (adler32-zlib) [ 4104.720638] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4104.831675] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 4106.424163] Key type lgssc registered [ 4106.864519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4109.940109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4109.949690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4111.165754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.878631] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 4116.684622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.865220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4118.649708] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 4121.842757] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4124.254902] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4127.057351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4127.140943] LustreError: 233920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4127.149666] LustreError: 233920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4127.213910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.510920] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_default_debug -1 all [ 4132.667700] Lustre: DEBUG MARKER: oleg650-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4136.427935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 4146.658985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.659883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4146.659991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4146.664360] Lustre: Skipped 1 previous similar message [ 4149.336061] Lustre: server umount lustre-OST0000 complete [ 4151.777577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.783620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.786311] Lustre: Skipped 2 previous similar messages [ 4156.897408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4156.899727] Lustre: Skipped 1 previous similar message [ 4162.017409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.019540] Lustre: Skipped 1 previous similar message [ 4165.088095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4165.148766] Lustre: server umount lustre-MDT0000 complete [ 4166.369171] LustreError: 232081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870158 with bad export cookie 10355070443445274134 [ 4166.372264] LustreError: MGC192.168.206.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4166.374382] LustreError: 232081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4166.487684] Lustre: server umount lustre-MDT0001 complete [ 4170.412116] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 4171.488381] Key type lgssc unregistered [ 4171.620422] LNet: 235789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4171.624324] LNetError: 235789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4171.632306] LNet: Removed LNI 192.168.206.150@tcp [ 4171.979385] Key type .llcrypt unregistered [ 4171.981367] Key type ._llcrypt unregistered [ 4181.794069] Key type ._llcrypt registered [ 4181.795700] Key type .llcrypt registered [ 4181.843788] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 4182.385937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4182.402533] alg: No test for adler32 (adler32-zlib) [ 4183.276790] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4183.376227] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 4184.968131] Key type lgssc registered [ 4185.394513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4452.291657] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 4453.408606] Key type lgssc unregistered [ 4453.547580] LNet: 237784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4453.550897] LNetError: 237784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4453.559403] LNet: Removed LNI 192.168.206.150@tcp [ 4453.907318] Key type .llcrypt unregistered [ 4453.908577] Key type ._llcrypt unregistered [ 4464.313931] Key type ._llcrypt registered [ 4464.315633] Key type .llcrypt registered [ 4464.364799] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing set_hostid [ 4471.588927] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing load_modules_local [ 4472.087067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4472.095488] alg: No test for adler32 (adler32-zlib) [ 4472.956256] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4473.063102] LNet: Added LNI 192.168.206.150@tcp [8/256/0/180] [ 4474.648195] Key type lgssc registered [ 4475.056751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4487.320229] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4357 sec ========== 08:27:59 (1788870479) [ 4488.064463] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:28:00 (1788870480) === [ 4489.513459] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:28:01 (1788870481) === [ 4506.084151] Lustre: DEBUG MARKER: oleg650-server.virtnet: executing unload_modules_local [ 4507.261252] Key type lgssc unregistered [ 4507.412934] LNet: 242985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4507.416625] LNetError: 242985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4507.424347] LNet: Removed LNI 192.168.206.150@tcp [ 4507.779448] Key type .llcrypt unregistered [ 4507.781254] Key type ._llcrypt unregistered