[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511904255 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003315] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008548] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009012] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012121] LSM: Security Framework initializing [ 0.013049] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015061] *** VALIDATE selinux *** [ 0.024011] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029704] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032113] *** VALIDATE tmpfs *** [ 0.034258] *** VALIDATE proc *** [ 0.035218] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044618] debug: unmapping init [mem 0xffffffffb9459000-0xffffffffb9460fff] [ 0.046161] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047601] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048019] ... version: 2 [ 0.049009] ... bit width: 48 [ 0.050000] ... generic registers: 4 [ 0.050999] ... value mask: 0000ffffffffffff [ 0.052009] ... max period: 00007fffffffffff [ 0.053010] ... fixed-purpose events: 3 [ 0.054008] ... event mask: 000000070000000f [ 0.055255] rcu: Hierarchical SRCU implementation. [ 0.057360] smp: Bringing up secondary CPUs ... [ 0.058580] x86: Booting SMP configuration: [ 0.059020] .... node #0, CPUs: #1 #2 #3 [ 0.062442] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136625] node 0 deferred pages initialised in 69ms [ 0.140587] devtmpfs: initialized [ 0.141228] x86/mm: Memory block size: 128MB [ 0.143979] gcov: version magic: 0x41383552 [ 0.145500] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147240] pinctrl core: initialized pinctrl subsystem [ 0.148217] [ 0.148833] ************************************************************* [ 0.149015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150016] ** ** [ 0.151014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152015] ** ** [ 0.153014] ** This means that this kernel is built to expose internal ** [ 0.154011] ** IOMMU data structures, which may compromise security on ** [ 0.155013] ** your system. ** [ 0.156015] ** ** [ 0.157014] ** If you see this message and you are not debugging the ** [ 0.158016] ** kernel, report this immediately to your vendor! ** [ 0.159016] ** ** [ 0.160013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161016] ************************************************************* [ 0.163023] NET: Registered protocol family 16 [ 0.164697] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165121] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168000] cpuidle: using governor menu [ 0.172152] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173736] PCI: Using configuration type 1 for base access [ 0.174129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202032] cryptd: max_cpu_qlen set to 1000 [ 0.208243] ACPI: Added _OSI(Module Device) [ 0.209013] ACPI: Added _OSI(Processor Device) [ 0.210054] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.211019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219861] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224696] ACPI: Interpreter enabled [ 0.225057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.226014] ACPI: Using IOAPIC for interrupt routing [ 0.227087] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228491] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240145] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243214] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245350] acpiphp: Slot [2] registered [ 0.246146] acpiphp: Slot [5] registered [ 0.247118] acpiphp: Slot [6] registered [ 0.248130] acpiphp: Slot [7] registered [ 0.249119] acpiphp: Slot [8] registered [ 0.250102] acpiphp: Slot [9] registered [ 0.251223] acpiphp: Slot [10] registered [ 0.253000] acpiphp: Slot [3] registered [ 0.253117] acpiphp: Slot [4] registered [ 0.254109] acpiphp: Slot [11] registered [ 0.255098] acpiphp: Slot [12] registered [ 0.256118] acpiphp: Slot [13] registered [ 0.257105] acpiphp: Slot [14] registered [ 0.258113] acpiphp: Slot [15] registered [ 0.259112] acpiphp: Slot [16] registered [ 0.260118] acpiphp: Slot [17] registered [ 0.261165] acpiphp: Slot [18] registered [ 0.262770] acpiphp: Slot [19] registered [ 0.263112] acpiphp: Slot [20] registered [ 0.264248] acpiphp: Slot [21] registered [ 0.265132] acpiphp: Slot [22] registered [ 0.266091] acpiphp: Slot [23] registered [ 0.267110] acpiphp: Slot [24] registered [ 0.268104] acpiphp: Slot [25] registered [ 0.269229] acpiphp: Slot [26] registered [ 0.270253] acpiphp: Slot [27] registered [ 0.271133] acpiphp: Slot [28] registered [ 0.272287] acpiphp: Slot [29] registered [ 0.273126] acpiphp: Slot [30] registered [ 0.274125] acpiphp: Slot [31] registered [ 0.275062] PCI host bridge to bus 0000:00 [ 0.276018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.279022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.280022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284318] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286887] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.296017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299970] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301635] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.302016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.304000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.311056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.316026] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382289] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.414014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.475087] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.529014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.559136] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586038] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.610027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.656024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.675917] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.695021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.717017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.742000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.760000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.774025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.784018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.816026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.837861] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.847015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.856015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.875026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.906228] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.915481] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.922235] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.926574] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.934640] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.947036] iommu: Default domain type: Passthrough [ 0.952754] SCSI subsystem initialized [ 0.956325] ACPI: bus type USB registered [ 0.966176] usbcore: registered new interface driver usbfs [ 0.976000] usbcore: registered new interface driver hub [ 0.983128] usbcore: registered new device driver usb [ 0.986214] pps_core: LinuxPPS API ver. 1 registered [ 0.988016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.998085] PTP clock support registered [ 1.009779] EDAC MC: Ver: 3.0.0 [ 1.014597] PCI: Using ACPI for IRQ routing [ 1.016892] NetLabel: Initializing [ 1.018010] NetLabel: domain hash size = 128 [ 1.027016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.036151] NetLabel: unlabeled traffic allowed by default [ 1.042357] vgaarb: loaded [ 1.045495] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.048071] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.058394] clocksource: Switched to clocksource kvm-clock [ 1.273677] VFS: Disk quotas dquot_6.6.0 [ 1.276813] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.281364] *** VALIDATE ramfs *** [ 1.283641] *** VALIDATE hugetlbfs *** [ 1.290240] pnp: PnP ACPI init [ 1.295660] pnp: PnP ACPI: found 6 devices [ 1.370317] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.378584] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.383510] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.388371] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.392294] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.396561] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.401219] NET: Registered protocol family 2 [ 1.405624] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.418522] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.429296] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.440887] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.447420] TCP: Hash tables configured (established 65536 bind 65536) [ 1.451759] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.457083] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.462407] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.465801] NET: Registered protocol family 1 [ 1.476043] RPC: Registered named UNIX socket transport module. [ 1.478710] RPC: Registered udp transport module. [ 1.485408] RPC: Registered tcp transport module. [ 1.487972] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.491909] NET: Registered protocol family 44 [ 1.493609] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.497082] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.501651] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.508956] PCI: CLS 0 bytes, default 64 [ 1.517884] Unpacking initramfs... [ 5.021590] debug: unmapping init [mem 0xffff9b68bcc54000-0xffff9b68bffbffff] [ 5.038580] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.043551] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 5.052560] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.732751] Initialise system trusted keyrings [ 6.736041] Key type blacklist registered [ 6.739356] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.753098] zbud: loaded [ 6.756788] *** VALIDATE nfs *** [ 6.758783] *** VALIDATE nfs4 *** [ 6.760786] pstore: using deflate compression [ 6.765607] Platform Keyring initialized [ 6.960136] NET: Registered protocol family 38 [ 6.963140] Key type asymmetric registered [ 6.965602] Asymmetric key parser 'x509' registered [ 6.968886] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.973572] io scheduler mq-deadline registered [ 6.977501] io scheduler kyber registered [ 6.982109] io scheduler bfq registered [ 6.985375] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.992435] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.998217] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.004146] ACPI: Power Button [PWRF] [ 7.017038] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.029810] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.052179] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.065078] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.091689] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.133705] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.178177] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.186593] Non-volatile memory driver v1.3 [ 7.191393] Linux agpgart interface v0.103 [ 7.240625] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 7.246293] vda: detected capacity change from 0 to 75087872 [ 7.280558] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.294263] vdb: detected capacity change from 0 to 1073741824 [ 7.338239] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.341927] vdc: detected capacity change from 0 to 2621440000 [ 7.367094] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.371097] vdd: detected capacity change from 0 to 2621440000 [ 7.394815] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.399252] vde: detected capacity change from 0 to 4294967296 [ 7.432389] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.437348] vdf: detected capacity change from 0 to 4294967296 [ 7.455424] libphy: Fixed MDIO Bus: probed [ 7.474148] usbcore: registered new interface driver usbserial_generic [ 7.476980] usbserial: USB Serial support registered for generic [ 7.486381] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.509743] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.525679] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.546379] mousedev: PS/2 mouse device common for all mice [ 7.554862] rtc_cmos 00:05: RTC can wake from S4 [ 7.564317] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.567394] rtc_cmos 00:05: registered as rtc0 [ 7.572279] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.575194] intel_pstate: CPU model not supported [ 7.579634] hid: raw HID events driver (C) Jiri Kosina [ 7.582770] usbcore: registered new interface driver usbhid [ 7.589191] usbhid: USB HID core driver [ 7.590808] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.594241] drop_monitor: Initializing network drop monitor service [ 7.609250] Initializing XFRM netlink socket [ 7.611086] NET: Registered protocol family 10 [ 7.619424] Segment Routing with IPv6 [ 7.623760] NET: Registered protocol family 17 [ 7.629919] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.630839] mpls_gso: MPLS GSO support [ 7.645508] RAS: Correctable Errors collector initialized. [ 7.648133] AVX version of gcm_enc/dec engaged. [ 7.649906] AES CTR mode by8 optimization enabled [ 7.815689] sched_clock: Marking stable (7815670776, 0)->(9095908909, -1280238133) [ 7.823842] registered taskstats version 1 [ 7.826648] Loading compiled-in X.509 certificates [ 7.828848] zswap: loaded using pool lzo/zbud [ 7.902306] Key type big_key registered [ 7.932440] Key type encrypted registered [ 7.938727] ima: No TPM chip found, activating TPM-bypass! [ 7.943678] ima: Allocated hash algorithm: sha1 [ 7.947261] ima: No architecture policies found [ 7.949994] evm: Initialising EVM extended attributes: [ 7.953362] evm: security.selinux [ 7.957885] evm: security.ima [ 7.959304] evm: security.capability [ 7.960866] evm: HMAC attrs: 0x1 [ 7.964621] rtc_cmos 00:05: setting system clock to 2026-08-27 11:19:42 UTC (1787829582) [ 7.977750] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 7.983578] debug: unmapping init [mem 0xffffffffb9182000-0xffffffffb9458fff] [ 8.001226] Write protecting the kernel read-only data: 28672k [ 8.007305] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 8.012578] debug: unmapping init [mem 0xffffffffb8114000-0xffffffffb81fffff] [ 8.100474] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.127282] systemd[1]: Detected virtualization kvm. [ 8.130612] systemd[1]: Detected architecture x86-64. [ 8.133629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.201798] systemd[1]: No hostname configured. [ 8.205238] systemd[1]: Set hostname to . [ 8.210946] random: systemd: uninitialized urandom read (16 bytes read) [ 8.219324] systemd[1]: Initializing machine ID from random generator. [ 8.392239] random: ln: uninitialized urandom read (6 bytes read) [ 8.604418] random: systemd: uninitialized urandom read (16 bytes read) [ 8.606956] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.616939] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 8.623465] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.186975] device-mapper: uevent: version 1.0.3 [ 10.190815] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.943074] virtio_net virtio0 ens2: renamed from eth0 [ 11.998691] random: fast init done [ 12.266238] scsi host0: ata_piix [ 12.411559] scsi host1: ata_piix [ 12.414765] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.420332] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.477164] random: crng init done [ 17.481567] random: 7 urandom warning(s) missed due to ratelimiting [ 18.217779] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.055186] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.042370] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.758766] SELinux: Disabled at runtime. [ 23.961352] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.973975] systemd[1]: Detected virtualization kvm. [ 23.987927] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.078913] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.099183] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.126941] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.146174] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.158868] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.177194] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.201985] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ 26.814886] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.641725] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.604078] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.692562] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.265051] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.317654] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.996876] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.839205] NFS: Registering the id_resolver key type [ 35.842412] Key type id_resolver registered [ 35.844455] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg657-server login: [ 100.064474] libcfs: loading out-of-tree module taints kernel. [ 100.151905] Key type ._llcrypt registered [ 100.155967] Key type .llcrypt registered [ 100.367264] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 116.451127] hrtimer: interrupt took 6118565 ns [ 118.882766] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 120.547422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.566113] alg: No test for adler32 (adler32-zlib) [ 122.057569] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 122.799480] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 124.569758] Key type lgssc registered [ 126.618909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.524314] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.886257] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 201.032817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 201.096559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.432044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.478816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.607279] Lustre: lustre-MDT0000: new disk, initializing [ 202.728204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.765383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.926814] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.127143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.336748] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.381160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.386928] Lustre: Skipped 1 previous similar message [ 223.446574] Lustre: lustre-MDT0001: new disk, initializing [ 223.501560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.528082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.534252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.827429] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.861269] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 243.003018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 243.304464] Lustre: lustre-OST0000: new disk, initializing [ 243.308565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 243.314949] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 243.402905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 249.934636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 249.951127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.060224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 250.277601] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.753168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 268.003714] Lustre: lustre-OST0001: new disk, initializing [ 268.012949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 268.028260] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 268.272220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 274.470792] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.017104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.036418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.122562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 286.236318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.214971] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.504495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing check_logdir /tmp/testlogs/ [ 305.076368] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing yml_node [ 309.361686] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 311.650390] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 314.376512] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 315.857651] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:24:50 EDT 2026 [ 332.227697] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 333.764393] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.713291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 351.717186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.718700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 351.754968] Lustre: Skipped 2 previous similar messages [ 355.081991] Lustre: server umount lustre-MDT0000 complete [ 358.426671] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829932 with bad export cookie 8188762116108244824 [ 358.439035] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.446640] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 361.956122] LustreError: 10102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.958907] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 361.993528] LustreError: 10102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 362.031018] Lustre: Skipped 1 previous similar message [ 362.040700] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.044789] Lustre: Skipped 4 previous similar messages [ 364.000929] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.002449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 364.026282] Lustre: Skipped 1 previous similar message [ 364.733767] Lustre: server umount lustre-MDT0001 complete [ 368.921525] Lustre: server umount lustre-OST0000 complete [ 372.466404] Lustre: server umount lustre-OST0001 complete [ 379.826395] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 390.554524] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 424.512827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.703543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.735275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 424.820593] Lustre: lustre-MDT0000: new disk, initializing [ 424.914555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.954493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 429.058560] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 440.078353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.252343] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.314592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 440.318694] Lustre: Skipped 1 previous similar message [ 440.392908] Lustre: lustre-MDT0001: new disk, initializing [ 440.543637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 440.596910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 440.614564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.149961] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 454.399347] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 460.148569] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.275368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.526558] Lustre: lustre-OST0000: new disk, initializing [ 467.530890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.539067] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.633356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.403715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.420272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.569333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 473.291286] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 484.899874] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.525361] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.683223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 493.999381] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.187304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.206412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.209484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.227261] Lustre: Skipped 2 previous similar messages [ 500.231907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.260406] Lustre: Skipped 1 previous similar message [ 502.385408] Lustre: server umount lustre-OST0000 complete [ 507.363809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 507.372289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.399292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 512.439279] Lustre: server umount lustre-MDT0000 complete [ 515.554830] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.585346] LustreError: 15091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 516.233258] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830090 with bad export cookie 8188762116108246357 [ 516.259130] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.586777] Lustre: server umount lustre-MDT0001 complete [ 519.994963] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 07:28:14 (1787830094) [ 526.318719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.692180] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.774088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 530.916706] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 531.939946] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.462571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.760074] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 552.462891] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.016048] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.919547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.732285] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 584.380644] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 593.889701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.901335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.928272] Lustre: Skipped 1 previous similar message [ 593.932246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.936683] Lustre: Skipped 2 previous similar messages [ 599.093784] Lustre: server umount lustre-OST0000 complete [ 605.153783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.183073] Lustre: Skipped 2 previous similar messages [ 609.435499] Lustre: server umount lustre-MDT0000 complete [ 610.273120] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.298487] LustreError: 19314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 613.611908] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830188 with bad export cookie 8188762116108247239 [ 613.624216] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.629677] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 614.219558] Lustre: server umount lustre-MDT0001 complete [ 628.651469] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 633.448994] Key type lgssc unregistered [ 633.807682] LNet: 22850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.824973] LNetError: 22850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.856402] LNet: Removed LNI 192.168.206.157@tcp [ 636.347170] Key type .llcrypt unregistered [ 636.366112] Key type ._llcrypt unregistered [ 660.872093] Key type ._llcrypt registered [ 660.873974] Key type .llcrypt registered [ 661.042649] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 661.947377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 661.965464] alg: No test for adler32 (adler32-zlib) [ 663.098263] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 663.399947] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 665.240187] Key type lgssc registered [ 666.427640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 674.453850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 674.490042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.949447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.771560] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 685.538538] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.557302] LustreError: 24266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 689.826220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.077289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 694.374523] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 702.211030] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 708.420797] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 715.534825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.151814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.278896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 723.265559] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 731.063476] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 749.598276] Lustre: server umount lustre-OST0000 complete [ 755.170903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.175526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.179718] Lustre: Skipped 1 previous similar message [ 755.190605] Lustre: Skipped 1 previous similar message [ 760.292360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.302567] Lustre: Skipped 1 previous similar message [ 763.498531] Lustre: server umount lustre-MDT0000 complete [ 765.423715] LustreError: 24262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.439994] LustreError: 24262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 766.436108] Lustre: 23576:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787830324/real 1787830324] req@0000000056266de6 x1874675878622208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787830340 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 766.462742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.493757] LustreError: 24261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.509956] LustreError: 24261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 767.844239] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830342 with bad export cookie 13862743029746849250 [ 767.882730] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.314560] Lustre: server umount lustre-MDT0001 complete [ 779.150945] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 781.951890] Key type lgssc unregistered [ 782.273654] LNet: 28245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.284632] LNetError: 28245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.310879] LNet: Removed LNI 192.168.206.157@tcp [ 783.301710] Key type .llcrypt unregistered [ 783.303644] Key type ._llcrypt unregistered [ 806.519566] Key type ._llcrypt registered [ 806.521704] Key type .llcrypt registered [ 806.652324] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 808.468531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.554648] alg: No test for adler32 (adler32-zlib) [ 809.743348] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 810.071813] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 811.736103] Key type lgssc registered [ 812.902317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 822.339649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 822.365749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.011518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 829.131610] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 833.521674] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.564307] LustreError: 29658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 838.627417] LustreError: 29659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.107103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.644479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 845.958603] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 853.487868] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 860.559780] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 876.287892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 876.942320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 885.457339] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 894.454249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 895.684755] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 904.677141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 904.686090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.707068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 904.711931] Lustre: Skipped 1 previous similar message [ 904.728378] Lustre: Skipped 1 previous similar message [ 908.864908] Lustre: server umount lustre-OST0000 complete [ 914.925378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 914.926725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.937506] Lustre: Skipped 1 previous similar message [ 918.930423] Lustre: server umount lustre-MDT0000 complete [ 920.038822] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.052401] LustreError: 29654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 922.346391] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830496 with bad export cookie 14654915133305143598 [ 922.350905] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 922.357738] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 922.775945] Lustre: server umount lustre-MDT0001 complete [ 933.529470] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 936.096564] Key type lgssc unregistered [ 936.381148] LNet: 33541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.388539] LNetError: 33541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.400870] LNet: Removed LNI 192.168.206.157@tcp [ 937.350815] Key type .llcrypt unregistered [ 937.352702] Key type ._llcrypt unregistered [ 960.083243] Key type ._llcrypt registered [ 960.091661] Key type .llcrypt registered [ 960.242249] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 961.866687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.910214] alg: No test for adler32 (adler32-zlib) [ 962.968353] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 963.445300] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 965.192238] Key type lgssc registered [ 966.230643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 974.630808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 974.645824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.147513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.456809] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 985.578583] LustreError: 34961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.600422] LustreError: 34961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 989.332099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.744755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.004416] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1001.094383] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1006.612913] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1013.690209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.051489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.965790] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1021.245510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1027.054730] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1040.868248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.886656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.908446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.892538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.904737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1041.906860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.937759] Lustre: Skipped 1 previous similar message [ 1043.326693] Lustre: server umount lustre-OST0000 complete [ 1047.011025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.032813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1047.037255] Lustre: Skipped 1 previous similar message [ 1052.129590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1052.139038] Lustre: Skipped 1 previous similar message [ 1052.912653] Lustre: server umount lustre-MDT0000 complete [ 1056.609471] LustreError: 36805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830631 with bad export cookie 9831093135485016677 [ 1056.616149] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1056.622417] LustreError: 36805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1057.022851] Lustre: server umount lustre-MDT0001 complete [ 1067.842103] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1070.872558] Key type lgssc unregistered [ 1071.259291] LNet: 38501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.264525] LNetError: 38501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.288754] LNet: Removed LNI 192.168.206.157@tcp [ 1072.372943] Key type .llcrypt unregistered [ 1072.374426] Key type ._llcrypt unregistered [ 1096.260620] Key type ._llcrypt registered [ 1096.264322] Key type .llcrypt registered [ 1096.350828] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1097.446242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1097.499774] alg: No test for adler32 (adler32-zlib) [ 1098.560483] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1098.813259] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1100.456378] Key type lgssc registered [ 1101.720436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.629419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.672141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.279153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.048798] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1120.758333] LustreError: 39916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.783660] LustreError: 39916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1125.408451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.978278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1130.361668] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1136.915765] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.539592] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1151.870708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1152.169129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.312331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1159.604585] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1168.519133] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1180.400729] Lustre: lustre-OST0000: Not available for connect from 192.168.206.57@tcp (stopping) [ 1181.159187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.168768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.181542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1181.196875] Lustre: Skipped 1 previous similar message [ 1181.225870] Lustre: Skipped 1 previous similar message [ 1183.668251] Lustre: server umount lustre-OST0000 complete [ 1205.737798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1205.739839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.747234] Lustre: Skipped 1 previous similar message [ 1205.757551] Lustre: Skipped 1 previous similar message [ 1209.729754] Lustre: server umount lustre-MDT0000 complete [ 1210.858959] LustreError: 39916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.883220] LustreError: 39916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1213.438490] LustreError: 39895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830787 with bad export cookie 7113992082805044151 [ 1213.442726] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.454769] LustreError: 39895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1219.855189] Lustre: server umount lustre-MDT0001 complete [ 1229.807620] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1232.603450] Key type lgssc unregistered [ 1232.991819] LNet: 43625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.996643] LNetError: 43625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.025231] LNet: Removed LNI 192.168.206.157@tcp [ 1233.995704] Key type .llcrypt unregistered [ 1233.999730] Key type ._llcrypt unregistered [ 1260.990254] Key type ._llcrypt registered [ 1260.992731] Key type .llcrypt registered [ 1261.152208] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1261.819678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1261.828938] alg: No test for adler32 (adler32-zlib) [ 1263.018856] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1263.278316] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1264.968151] Key type lgssc registered [ 1266.216335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1273.844070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1273.861728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.183256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.005728] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1284.578542] LustreError: 45041:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.619572] LustreError: 45041:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1288.617320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.871353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1292.768741] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1300.160803] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1305.215242] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1313.251225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.595684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.733684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1321.717212] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1329.915765] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1336.356494] Lustre: server umount lustre-MDT0000 complete [ 1337.073518] LustreError: 45037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.298156] LustreError: 45020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830914 with bad export cookie 3708243590100309415 [ 1340.306552] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.314987] LustreError: 45020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1340.388386] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.390191] LustreError: 46934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.403047] Lustre: Skipped 1 previous similar message [ 1340.422674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1342.191403] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.57@tcp (stopping) [ 1345.508790] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1345.509866] LustreError: 45037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.550684] LustreError: 45037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1346.851368] Lustre: server umount lustre-MDT0001 complete [ 1367.277583] Lustre: server umount lustre-OST0000 complete [ 1382.772392] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1385.358510] Key type lgssc unregistered [ 1385.690317] LNet: 48903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.698521] LNetError: 48903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.728717] LNet: Removed LNI 192.168.206.157@tcp [ 1386.615361] Key type .llcrypt unregistered [ 1386.617758] Key type ._llcrypt unregistered [ 1409.526738] Key type ._llcrypt registered [ 1409.530202] Key type .llcrypt registered [ 1409.644283] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1410.969842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1411.043267] alg: No test for adler32 (adler32-zlib) [ 1412.068524] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1412.318700] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1414.048160] Key type lgssc registered [ 1415.402859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1423.603155] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1423.620604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.074429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1444.192410] LustreError: 50290:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 1470.816194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1477.535074] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1485.875366] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1651.272246] Lustre: server umount lustre-OST0000 complete [ 1664.794619] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1667.314988] Key type lgssc unregistered [ 1667.539613] LNet: 51967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.546193] LNetError: 51967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.564398] LNet: Removed LNI 192.168.206.157@tcp [ 1668.313833] Key type .llcrypt unregistered [ 1668.319621] Key type ._llcrypt unregistered [ 1691.406352] Key type ._llcrypt registered [ 1691.410141] Key type .llcrypt registered [ 1691.581184] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1692.816302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.092047] alg: No test for adler32 (adler32-zlib) [ 1694.147980] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1694.462319] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1696.232175] Key type lgssc registered [ 1697.255764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.599867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1704.614912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.974198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.664508] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1716.197674] LustreError: 53385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.215625] LustreError: 53385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1717.753068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.152937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1722.108880] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1729.040753] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1734.071462] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1740.677581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.941569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.958455] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1751.256782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1752.204628] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.650659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.651402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.660860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.670880] Lustre: Skipped 1 previous similar message [ 1760.931637] Lustre: server umount lustre-OST0000 complete [ 1764.322467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1764.331567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.345150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1764.349942] Lustre: Skipped 1 previous similar message [ 1766.882621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.884694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.687536] Lustre: server umount lustre-MDT0000 complete [ 1772.001194] LustreError: 54441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.031771] LustreError: 54441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1774.932144] LustreError: 53364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831349 with bad export cookie 10247444696033407236 [ 1774.939794] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.953876] LustreError: 53364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1775.382436] Lustre: server umount lustre-MDT0001 complete [ 1785.959370] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1789.066639] Key type lgssc unregistered [ 1789.431697] LNet: 56921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.436397] LNetError: 56921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.518547] LNet: Removed LNI 192.168.206.157@tcp [ 1791.185990] Key type .llcrypt unregistered [ 1791.189329] Key type ._llcrypt unregistered [ 1813.375689] Key type ._llcrypt registered [ 1813.382229] Key type .llcrypt registered [ 1813.470291] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 1814.695302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.752436] alg: No test for adler32 (adler32-zlib) [ 1815.880211] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 1816.204881] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 1817.920155] Key type lgssc registered [ 1818.944650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1826.296040] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1826.320201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1841.890066] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1847.008377] LustreError: 58308:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 1873.696297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.707593] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1884.133743] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1889.843734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.249890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.311608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1893.751962] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1895.400076] LustreError: 58333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.411988] LustreError: 58333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1900.516547] LustreError: 58334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.412415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.655310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.924741] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 1913.581317] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.406229] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1922.017818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.025158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.048795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.043645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.047951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1923.076246] Lustre: Skipped 1 previous similar message [ 1927.365980] Lustre: server umount lustre-OST0000 complete [ 1949.161706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.176185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1949.182948] Lustre: Skipped 1 previous similar message [ 1952.162666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1952.165414] Lustre: Skipped 1 previous similar message [ 1955.482787] Lustre: server umount lustre-MDT0000 complete [ 1957.346903] LustreError: 59348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.377385] LustreError: 59348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1959.118410] LustreError: 58316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831533 with bad export cookie 17929403838896068336 [ 1959.126129] LustreError: 58316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1959.126417] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.455245] Lustre: server umount lustre-MDT0001 complete [ 1975.016320] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 1977.267163] Key type lgssc unregistered [ 1977.532594] LNet: 62057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.539193] LNetError: 62057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.574020] LNet: Removed LNI 192.168.206.157@tcp [ 1978.380260] Key type .llcrypt unregistered [ 1978.382446] Key type ._llcrypt unregistered [ 2001.806542] Key type ._llcrypt registered [ 2001.809196] Key type .llcrypt registered [ 2001.927270] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2002.831230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.975958] alg: No test for adler32 (adler32-zlib) [ 2004.013955] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2004.175901] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2005.864482] Key type lgssc registered [ 2007.093140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2014.157256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2014.185509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2015.506638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2020.049943] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2025.954791] LustreError: 63478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2025.982140] LustreError: 63478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2028.799399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2029.119418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2034.115960] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2041.620396] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2048.097378] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2055.536987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.872483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.982634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2062.013391] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2069.337055] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2086.882810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2086.887753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.896927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.969185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.973426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2089.985051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.055988] Lustre: server umount lustre-OST0000 complete [ 2100.203101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.204359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.225584] Lustre: Skipped 1 previous similar message [ 2100.237783] Lustre: Skipped 2 previous similar messages [ 2102.023530] Lustre: server umount lustre-MDT0000 complete [ 2105.313709] LustreError: 63477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.331690] LustreError: 63477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2105.628590] LustreError: 63457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831680 with bad export cookie 5006843647261097357 [ 2105.633274] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.640838] LustreError: 63457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2106.079655] Lustre: server umount lustre-MDT0001 complete [ 2116.213871] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2118.728301] Key type lgssc unregistered [ 2118.951706] LNet: 67036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.956727] LNetError: 67036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.974710] LNet: Removed LNI 192.168.206.157@tcp [ 2119.705614] Key type .llcrypt unregistered [ 2119.711325] Key type ._llcrypt unregistered [ 2140.355959] Key type ._llcrypt registered [ 2140.358564] Key type .llcrypt registered [ 2140.477363] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2141.152565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.200591] alg: No test for adler32 (adler32-zlib) [ 2142.188997] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2142.415938] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2144.056200] Key type lgssc registered [ 2144.972594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2152.994349] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 07:55:27 (1787831727) [ 2158.873792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2158.902904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.324525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2164.264721] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2169.830386] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.846775] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2172.035795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2172.395218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.897539] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2184.275234] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2190.532024] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2197.293309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.710255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.484296] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2204.925156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2211.212120] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2220.035959] Lustre: Failing over lustre-MDT0000 [ 2220.280905] Lustre: server umount lustre-MDT0000 complete [ 2222.576582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.579781] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.587852] Lustre: Skipped 1 previous similar message [ 2222.626284] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2227.687725] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.713894] LustreError: 68720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2229.513936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.672605] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.867365] Lustre: *** cfs_fail_loc=135, val=0*** [ 2229.972651] LustreError: 71810:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2229.987522] LustreError: 71810:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -2 [ 2230.003668] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2230.003668] [ 2230.019410] LustreError: MGC192.168.206.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2230.032985] LustreError: 71798:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2230.052524] LustreError: 71798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2230.060268] LustreError: 71798:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2230.071244] LustreError: 71798:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2230.096515] Lustre: server umount lustre-MDT0000 complete [ 2230.101125] LustreError: 71798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2232.816532] LustreError: 69733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.840355] LustreError: 69733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2237.584782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.719988] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.961848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.029596] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2238.681802] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2242.600710] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2243.051247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2243.070544] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2243.121494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2253.298122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.332601] Lustre: Skipped 1 previous similar message [ 2253.338927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.343330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.346079] Lustre: Skipped 1 previous similar message [ 2256.102866] Lustre: server umount lustre-OST0000 complete [ 2263.521648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.526481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.535589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2263.558657] Lustre: Skipped 2 previous similar messages [ 2263.575242] Lustre: Skipped 1 previous similar message [ 2266.327218] Lustre: server umount lustre-MDT0000 complete [ 2268.641121] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.674364] LustreError: 68721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2270.812217] LustreError: 68701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831845 with bad export cookie 16600663228901287828 [ 2270.827358] LustreError: 68701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2270.827393] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.154043] Lustre: server umount lustre-MDT0001 complete [ 2284.779612] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2288.302296] Key type lgssc unregistered [ 2288.729453] LNet: 73589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.761250] LNetError: 73589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.789724] LNet: Removed LNI 192.168.206.157@tcp [ 2289.623616] Key type .llcrypt unregistered [ 2289.631237] Key type ._llcrypt unregistered [ 2312.230040] Key type ._llcrypt registered [ 2312.231873] Key type .llcrypt registered [ 2312.316572] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2313.167404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.189158] alg: No test for adler32 (adler32-zlib) [ 2314.275809] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2314.506700] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2316.184152] Key type lgssc registered [ 2317.331654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.331984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2325.361721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.772783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.613990] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2337.250108] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.261922] LustreError: 75002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2340.600130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.860954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2345.253325] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2353.464658] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2359.276458] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2365.248655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2365.536316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2371.164228] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2371.702130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2377.710036] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2382.355960] Lustre: Failing over lustre-MDT0000 [ 2382.615854] Lustre: server umount lustre-MDT0000 complete [ 2384.632688] LustreError: 76245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.923813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2386.932787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.941183] LustreError: 75731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.717903] LustreError: 74999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.736664] LustreError: 74999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2390.720669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.907490] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.069850] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2391.190699] LustreError: 78091:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2391.193648] LustreError: 78091:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -9 [ 2391.200075] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2391.200075] [ 2391.204258] LustreError: MGC192.168.206.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2391.214419] LustreError: 78079:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2391.220713] LustreError: 78079:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2391.224767] LustreError: 78079:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2391.228447] LustreError: 78079:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2391.255400] Lustre: server umount lustre-MDT0000 complete [ 2391.259315] LustreError: 78079:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2394.840183] LustreError: 76245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.866701] LustreError: 76245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2397.246268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.401285] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.413142] LustreError: 74994:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2397.710607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.770322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2399.958883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2401.799726] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2402.792965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2402.815935] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2402.839928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2413.029898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.030550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.044698] Lustre: Skipped 2 previous similar messages [ 2413.060505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2414.446741] Lustre: server umount lustre-OST0000 complete [ 2418.155660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2418.160478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2418.164890] LustreError: Skipped 1 previous similar message [ 2418.165829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2418.165835] Lustre: Skipped 1 previous similar message [ 2418.200416] Lustre: Skipped 2 previous similar messages [ 2423.266116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2423.273337] Lustre: Skipped 2 previous similar messages [ 2424.239678] Lustre: server umount lustre-MDT0000 complete [ 2427.737813] LustreError: 74983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832002 with bad export cookie 16564561447075915279 [ 2427.739057] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.749723] LustreError: 74983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2428.207921] Lustre: server umount lustre-MDT0001 complete [ 2439.916222] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2442.600449] Key type lgssc unregistered [ 2442.875700] LNet: 79872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2442.880350] LNetError: 79872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2442.902318] LNet: Removed LNI 192.168.206.157@tcp [ 2443.671552] Key type .llcrypt unregistered [ 2443.673229] Key type ._llcrypt unregistered [ 2455.269844] Key type ._llcrypt registered [ 2455.271778] Key type .llcrypt registered [ 2455.359958] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2455.641317] Key type .llcrypt unregistered [ 2455.642948] Key type ._llcrypt unregistered [ 2476.563390] Key type ._llcrypt registered [ 2476.566063] Key type .llcrypt registered [ 2476.650730] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2477.570847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2477.587153] alg: No test for adler32 (adler32-zlib) [ 2478.658456] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2478.864957] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2480.505812] Key type lgssc registered [ 2481.569657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2487.974538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2488.001771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.363179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.935837] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2498.534487] LustreError: 81792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.548187] LustreError: 81792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2500.201223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.455586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2504.249554] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2509.588258] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2514.293939] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.749318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.092621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.909281] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2533.643167] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2534.461129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2549.735042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.739441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.756891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2552.468928] Lustre: server umount lustre-OST0000 complete [ 2556.897169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2556.904737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.925980] Lustre: Skipped 1 previous similar message [ 2556.930758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.936554] Lustre: Skipped 1 previous similar message [ 2559.970678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.973084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.900881] Lustre: server umount lustre-MDT0000 complete [ 2565.099261] LustreError: 83915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.122517] LustreError: 83915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2565.141541] LustreError: 81773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832139 with bad export cookie 944034694584025485 [ 2565.148483] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.151909] LustreError: 81773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2565.382487] Lustre: server umount lustre-MDT0001 complete [ 2576.411577] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2579.788862] Key type lgssc unregistered [ 2580.059746] LNet: 85344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.066464] LNetError: 85344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.084881] LNet: Removed LNI 192.168.206.157@tcp [ 2580.901752] Key type .llcrypt unregistered [ 2580.904358] Key type ._llcrypt unregistered [ 2604.064318] Key type ._llcrypt registered [ 2604.067382] Key type .llcrypt registered [ 2604.161138] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2605.032575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.140509] alg: No test for adler32 (adler32-zlib) [ 2606.152301] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2606.391902] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2608.065978] Key type lgssc registered [ 2609.301584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.754260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2615.787731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.173406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.306457] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2626.531297] LustreError: 86757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.551895] LustreError: 86757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2629.147280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.496957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2633.532928] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2640.451742] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2646.157414] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.290850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.626244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.973297] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2661.879100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2664.185326] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2672.100942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.107890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.121910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.133104] Lustre: Skipped 1 previous similar message [ 2675.748619] Lustre: server umount lustre-OST0000 complete [ 2680.803775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2680.811903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.824856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.833482] Lustre: Skipped 1 previous similar message [ 2682.338859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.344981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.521249] Lustre: server umount lustre-MDT0000 complete [ 2687.458966] LustreError: 86757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.494663] LustreError: 86757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2689.381702] LustreError: 86738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832263 with bad export cookie 15032254695933185885 [ 2689.403729] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.877198] Lustre: server umount lustre-MDT0001 complete [ 2701.744915] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2704.945364] Key type lgssc unregistered [ 2705.172685] LNet: 90290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.178565] LNetError: 90290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.197795] LNet: Removed LNI 192.168.206.157@tcp [ 2706.020226] Key type .llcrypt unregistered [ 2706.022070] Key type ._llcrypt unregistered [ 2726.396160] Key type ._llcrypt registered [ 2726.398801] Key type .llcrypt registered [ 2726.553157] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2727.299207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2727.377549] alg: No test for adler32 (adler32-zlib) [ 2728.445848] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2728.691449] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2730.352218] Key type lgssc registered [ 2731.394169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.952778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2739.971957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.478467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.976560] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2750.952257] LustreError: 91682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.980725] LustreError: 91682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2756.039746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.067181] LustreError: 91681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.595909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.174736] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2770.368317] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2776.774943] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2784.484646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.789894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.847817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2791.285774] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2799.017066] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.485661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.494159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.505464] LustreError: Skipped 1 previous similar message [ 2816.507139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2816.531248] Lustre: Skipped 1 previous similar message [ 2821.609207] LustreError: 93505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.746311] Lustre: server umount lustre-OST0000 complete [ 2826.731566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.736097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2826.744178] Lustre: Skipped 1 previous similar message [ 2826.751488] Lustre: Skipped 2 previous similar messages [ 2831.841580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.126774] Lustre: server umount lustre-MDT0000 complete [ 2836.158041] LustreError: 91662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832410 with bad export cookie 10341068033887535054 [ 2836.173755] LustreError: 91662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2836.185581] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.543749] Lustre: server umount lustre-MDT0001 complete [ 2848.670540] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 2852.255268] Key type lgssc unregistered [ 2852.642754] LNet: 95241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2852.658475] LNetError: 95241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2852.676850] LNet: Removed LNI 192.168.206.157@tcp [ 2853.605680] Key type .llcrypt unregistered [ 2853.607490] Key type ._llcrypt unregistered [ 2878.654318] Key type ._llcrypt registered [ 2878.663023] Key type .llcrypt registered [ 2878.866187] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 2880.591644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2880.925745] alg: No test for adler32 (adler32-zlib) [ 2881.906123] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 2882.092572] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 2883.784165] Key type lgssc registered [ 2885.228935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2893.941324] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2893.994327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2909.794570] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2914.913218] LustreError: 96627:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 2941.600204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.495878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2955.777721] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2962.265049] Lustre: server umount lustre-OST0000 complete [ 2971.895913] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 08:09:06 (1787832546) [ 2979.048220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.805591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.761822] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 2984.929330] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.959917] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2990.053473] LustreError: 98269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2994.987361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.178295] LustreError: 98268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.540627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.898572] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3008.720730] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3014.296314] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3021.170781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.286826] LustreError: 100089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3021.596077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.694143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3027.632796] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3035.242810] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3050.467858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3050.478133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.481949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3050.491655] Lustre: Skipped 1 previous similar message [ 3052.860836] Lustre: server umount lustre-OST0000 complete [ 3060.706650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.724740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.733115] Lustre: Skipped 1 previous similar message [ 3063.821235] Lustre: server umount lustre-MDT0000 complete [ 3065.830970] LustreError: 99241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.854892] LustreError: 99241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3068.481217] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832643 with bad export cookie 3198452102136348715 [ 3068.509250] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3068.512735] LustreError: 98249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3068.928089] Lustre: server umount lustre-MDT0001 complete [ 3080.089250] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 3083.255808] Key type lgssc unregistered [ 3083.715668] LNet: 101965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.724459] LNetError: 101965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.747515] LNet: Removed LNI 192.168.206.157@tcp [ 3084.636408] Key type .llcrypt unregistered [ 3084.638686] Key type ._llcrypt unregistered [ 3123.333269] Key type ._llcrypt registered [ 3123.335190] Key type .llcrypt registered [ 3123.413861] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3124.478601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3124.529498] alg: No test for adler32 (adler32-zlib) [ 3125.597312] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3125.789862] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 3127.440357] Key type lgssc registered [ 3128.564716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.058845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.080345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3138.553652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.591688] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3148.773582] LustreError: 104150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.801604] LustreError: 104150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3153.370115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.908855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.232111] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3165.148233] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3170.666069] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3177.111615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3177.389564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3182.724580] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3183.530922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3189.175374] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3196.362516] Lustre: server umount lustre-OST0000 complete [ 3199.975410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.993162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.028730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.064715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.070133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.191307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.201503] Lustre: Skipped 2 previous similar messages [ 3210.325884] Lustre: server umount lustre-MDT0000 complete [ 3214.308461] LustreError: 104881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.328560] LustreError: 104881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3214.392600] LustreError: 104130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832788 with bad export cookie 10335462528293266001 [ 3214.410288] LustreError: 104130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3214.417202] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.208455] Lustre: server umount lustre-MDT0001 complete [ 3236.223234] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 3238.789957] Key type lgssc unregistered [ 3239.099617] LNet: 107698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3239.109468] LNetError: 107698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.135240] LNet: Removed LNI 192.168.206.157@tcp [ 3240.002934] Key type .llcrypt unregistered [ 3240.009983] Key type ._llcrypt unregistered [ 3262.213629] Key type ._llcrypt registered [ 3262.219606] Key type .llcrypt registered [ 3262.360794] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3263.630126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3263.900699] alg: No test for adler32 (adler32-zlib) [ 3264.950639] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3265.198202] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 3266.866627] Key type lgssc registered [ 3268.182829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3275.718756] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3275.738175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.232870] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3296.352388] LustreError: 108943:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 3323.040217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.122794] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3334.457874] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3341.683887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.932390] LustreError: MGC192.168.206.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3341.945372] LustreError: 109967:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3341.951493] LustreError: 109967:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3341.960799] LustreError: 109967:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3341.973942] LustreError: 108950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832916 with bad export cookie 15682790470572994834 [ 3341.997176] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.061846] Lustre: server umount lustre-MDT0000 complete [ 3342.065536] LustreError: 109967:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3369.143701] Lustre: server umount lustre-OST0000 complete [ 3378.168765] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3387.704974] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3419.470320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.619447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3419.635580] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3419.684544] Lustre: lustre-MDT0000: new disk, initializing [ 3419.732471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.749763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3424.201970] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3435.312751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.458929] Lustre: 112759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3435.503210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3435.507284] Lustre: Skipped 1 previous similar message [ 3435.572686] Lustre: lustre-MDT0001: new disk, initializing [ 3435.666096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3435.704123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3435.733604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3440.764988] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3449.735400] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3455.486179] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3461.849735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3462.114386] Lustre: lustre-OST0000: new disk, initializing [ 3462.116814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3462.134669] Lustre: 114727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3462.219970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.629035] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3469.905896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3469.912054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3470.047962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3478.226251] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.134172] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3483.333389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3487.659502] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3487.857517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3490.788186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3490.789608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3490.790938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3490.797915] Lustre: Skipped 1 previous similar message [ 3495.749583] Lustre: server umount lustre-OST0000 complete [ 3501.028636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.056480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3501.062623] Lustre: Skipped 2 previous similar messages [ 3506.011182] Lustre: server umount lustre-MDT0000 complete [ 3509.512260] LustreError: 112750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833084 with bad export cookie 15682790470572994939 [ 3509.524268] LustreError: 112750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3509.530184] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3509.892325] Lustre: server umount lustre-MDT0001 complete [ 3518.998382] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 08:18:12 (1787833092) [ 3522.360604] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3540.443861] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3550.140763] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3584.523547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.723429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3584.764495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3584.827748] Lustre: lustre-MDT0000: new disk, initializing [ 3584.914708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.929589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.237694] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3599.147979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.272099] Lustre: 119628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3599.335488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3599.345197] Lustre: Skipped 1 previous similar message [ 3599.435401] Lustre: lustre-MDT0001: new disk, initializing [ 3599.500639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.532651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3599.547201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3603.519742] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3612.307758] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.238379] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3623.495319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.726209] Lustre: lustre-OST0000: new disk, initializing [ 3623.728884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3623.733751] Lustre: 121593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.809671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.413986] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3629.639571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3629.660606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3629.796886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3638.673828] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3642.644301] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3642.828205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3647.063546] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3647.209408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3650.532392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.544170] LustreError: Skipped 1 previous similar message [ 3650.549380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.562628] Lustre: Skipped 1 previous similar message [ 3650.570502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.662383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.670909] Lustre: Skipped 1 previous similar message [ 3660.776662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3660.788347] Lustre: Skipped 2 previous similar messages [ 3663.328138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3663.479265] Lustre: server umount lustre-OST0000 complete [ 3671.013067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.026691] Lustre: Skipped 1 previous similar message [ 3671.032301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3671.036300] Lustre: Skipped 1 previous similar message [ 3673.733988] Lustre: server umount lustre-MDT0000 complete [ 3676.128974] LustreError: 120658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.153106] LustreError: 120658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3677.127718] LustreError: 119619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833251 with bad export cookie 15682790470572995821 [ 3677.129813] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.145287] LustreError: 119619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3677.447270] Lustre: server umount lustre-MDT0001 complete [ 3684.935501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3685.384291] LustreError: 123810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.482617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3689.582159] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3690.467662] LustreError: 123811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.564154] LustreError: 123810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.169554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.393930] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3709.217345] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3715.383964] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3722.215849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3728.998211] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3735.977283] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3746.115796] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3752.429521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.431856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.441544] LustreError: Skipped 1 previous similar message [ 3752.449936] Lustre: Skipped 1 previous similar message [ 3752.454230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.321846] Lustre: server umount lustre-OST0000 complete [ 3762.660378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.672715] Lustre: Skipped 2 previous similar messages [ 3767.420749] Lustre: server umount lustre-MDT0000 complete [ 3767.777897] LustreError: 124860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3767.800600] LustreError: 124860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3770.809423] LustreError: 123790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833345 with bad export cookie 15682790470572996703 [ 3770.821281] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3770.825613] LustreError: 123790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3771.149494] Lustre: server umount lustre-MDT0001 complete [ 3781.226973] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 3784.124569] Key type lgssc unregistered [ 3784.413170] LNet: 127440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3784.420743] LNetError: 127440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3784.442769] LNet: Removed LNI 192.168.206.157@tcp [ 3785.395920] Key type .llcrypt unregistered [ 3785.406606] Key type ._llcrypt unregistered [ 3802.607678] Key type ._llcrypt registered [ 3802.613915] Key type .llcrypt registered [ 3802.698264] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 3816.220646] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 3817.178433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3817.282956] alg: No test for adler32 (adler32-zlib) [ 3818.345250] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 3818.642992] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 3820.336969] Key type lgssc registered [ 3821.711641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3853.539036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3853.596189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3854.872340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3854.889113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3854.942636] Lustre: lustre-MDT0000: new disk, initializing [ 3854.988937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3854.999955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3859.580286] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3869.866101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.990637] Lustre: 130593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3870.063552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3870.068047] Lustre: Skipped 1 previous similar message [ 3870.197202] Lustre: lustre-MDT0001: new disk, initializing [ 3870.336893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.377507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3870.392918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3874.405992] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3885.360356] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.898159] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3899.376140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.660814] Lustre: lustre-OST0000: new disk, initializing [ 3899.665568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3899.672874] Lustre: 132562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3899.783359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.183340] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3906.650117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3906.673801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3906.840386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3915.411429] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3919.384349] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3919.628983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3923.292878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3923.425733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3927.529028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.530976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3927.531416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.543310] Lustre: Skipped 1 previous similar message [ 3932.642478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.648903] Lustre: Skipped 2 previous similar messages [ 3937.777276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.787891] Lustre: Skipped 1 previous similar message [ 3939.296148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3939.517813] Lustre: server umount lustre-OST0000 complete [ 3946.981539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3946.989166] LustreError: Skipped 1 previous similar message [ 3946.993288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3947.006482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.002120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3953.122134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.134732] Lustre: Skipped 3 previous similar messages [ 3957.216346] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3957.495316] Lustre: server umount lustre-MDT0000 complete [ 3960.633810] LustreError: 130586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833535 with bad export cookie 2216315398421278353 [ 3960.641173] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3960.644838] LustreError: 130586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3960.960451] Lustre: server umount lustre-MDT0001 complete [ 3968.198923] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 08:25:42 (1787833542) [ 3973.620840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.063664] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.078357] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3974.149141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.277353] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3979.240121] LustreError: 134978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3984.353510] LustreError: 134977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.869356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.131743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3989.945834] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 3997.139220] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4002.748157] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4006.890317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4006.898863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.908237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4010.978775] LustreError: 136046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.009927] LustreError: 136046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4011.041550] Lustre: server umount lustre-MDT0000 complete [ 4014.052818] LustreError: 134959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833588 with bad export cookie 2216315398421279235 [ 4014.059864] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.071826] LustreError: 134959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4014.359166] Lustre: server umount lustre-MDT0001 complete [ 4021.218265] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 08:26:35 (1787833595) [ 4027.479627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4043.108887] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4048.224492] LustreError: 137391:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 4073.888674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.007662] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4087.535987] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4089.656555] Lustre: server umount lustre-OST0000 complete [ 4097.793328] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 08:27:51 (1787833671) [ 4105.217741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.219145] LustreError: 138837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.368428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.489220] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4115.937951] LustreError: 138837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.961273] LustreError: 138837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4120.609721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.310983] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4133.298882] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4139.266752] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4146.703234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4147.127967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.137214] Lustre: Skipped 1 previous similar message [ 4153.548883] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4161.145531] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4173.961507] Lustre: server umount lustre-MDT0000 complete [ 4175.847669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.860081] LustreError: 138837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.871921] Lustre: Skipped 3 previous similar messages [ 4175.898410] LustreError: 138837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4177.573255] LustreError: 138818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833752 with bad export cookie 2216315398421279893 [ 4177.576328] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.581661] LustreError: 138818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4177.949727] Lustre: server umount lustre-MDT0001 complete [ 4191.814911] Lustre: server umount lustre-OST0000 complete [ 4200.582994] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 08:29:34 (1787833774) [ 4208.276293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4208.713774] LustreError: 142508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.092273] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4222.156197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.591897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4222.600211] Lustre: Skipped 1 previous similar message [ 4227.473987] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4235.282750] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4241.429481] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4248.098053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.638165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4254.403443] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4262.556628] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4267.603969] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4267.830310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4272.991633] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4273.199875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4279.266313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4279.270757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.285182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4279.296984] Lustre: Skipped 1 previous similar message [ 4279.313362] Lustre: Skipped 3 previous similar messages [ 4281.563648] Lustre: server umount lustre-OST0000 complete [ 4288.995355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4292.050804] Lustre: server umount lustre-MDT0000 complete [ 4294.629642] LustreError: 143238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.638426] LustreError: 143238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4295.800862] LustreError: 142490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833870 with bad export cookie 2216315398421280971 [ 4295.808202] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.815648] LustreError: 142490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4296.147291] Lustre: server umount lustre-MDT0001 complete [ 4305.201156] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 08:31:19 (1787833879) [ 4310.812083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.368957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4331.488568] LustreError: 146670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 4357.216334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4357.227219] Lustre: Skipped 1 previous similar message [ 4362.976843] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4370.558356] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4377.207495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.698700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4381.900948] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4392.750881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.207398] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4406.303516] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4412.272477] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4416.796238] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4416.971954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4422.091635] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4422.332938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4429.288473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4429.297926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.299655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4429.319452] Lustre: Skipped 4 previous similar messages [ 4429.327889] Lustre: Skipped 4 previous similar messages [ 4430.692453] Lustre: server umount lustre-MDT0000 complete [ 4434.401768] LustreError: 147705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.413936] LustreError: 147705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4434.422694] LustreError: 146678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834008 with bad export cookie 2216315398421281993 [ 4434.434633] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.438224] LustreError: 146678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4434.767064] Lustre: server umount lustre-MDT0001 complete [ 4447.683598] Lustre: server umount lustre-OST0000 complete [ 4456.370560] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 08:33:50 (1787834030) [ 4463.546446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4479.200687] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4484.324832] LustreError: 150864:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 4515.907923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4523.205762] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.984853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.551267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4534.740724] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4544.224101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.049783] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4556.360298] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4561.714910] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4568.494351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4568.687744] Lustre: lustre-OST0001: new disk, initializing [ 4568.693426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4568.701168] Lustre: 153739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4574.450634] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4575.290917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4575.305339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4575.420941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4583.417791] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4587.749175] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4587.910422] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4592.744963] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4592.985303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.685337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.691707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.703896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.715342] Lustre: Skipped 1 previous similar message [ 4601.386465] Lustre: server umount lustre-OST0000 complete [ 4610.864417] Lustre: server umount lustre-OST0001 complete [ 4621.105784] Lustre: server umount lustre-MDT0000 complete [ 4624.951414] LustreError: 150870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834199 with bad export cookie 2216315398421282945 [ 4624.966505] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.396233] Lustre: server umount lustre-MDT0001 complete [ 4647.460760] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 08:37:01 (1787834221) [ 4649.198051] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4650.810402] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4652.563684] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 08:37:06 (1787834226) [ 4658.965372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.226970] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4659.251404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4659.387621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4659.391441] Lustre: Skipped 6 previous similar messages [ 4663.526726] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4671.302270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.378563] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4671.400203] Lustre: 157298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.069662] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4683.192475] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4688.312951] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4694.703044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.878736] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4701.019780] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4703.925085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4708.844698] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4713.548404] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4713.712918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4718.186578] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4718.413669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4724.707278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4724.711618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.718917] LustreError: Skipped 4 previous similar messages [ 4724.723407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4724.739891] Lustre: Skipped 6 previous similar messages [ 4724.751376] Lustre: Skipped 6 previous similar messages [ 4727.227357] Lustre: server umount lustre-OST0000 complete [ 4750.437276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4757.847146] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4765.175823] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.211214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.217154] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4765.254764] Lustre: Skipped 1 previous similar message [ 4766.750444] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.666142] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4771.845791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4776.322558] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4776.580972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4782.101377] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c8686d48000.ost_server_uuid 50 [ 4783.554880] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c8686d48000.ost_server_uuid in FULL state after 0 sec [ 4783.768774] LustreError: 159005:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.206.57@tcp) returned error from blocking AST (req@000000008c67a0d8 x1874679188158720 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000a6d44a41/0x1ec1ef1cc12109db lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.57@tcp remote: 0xeb28ae60c9b49292 expref: 5 pid: 157304 timeout: 4883 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4783.792136] LustreError: lustre-MDT0000: A client on nid 192.168.206.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4783.799314] LustreError: 157293:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.206.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000a6d44a41/0x1ec1ef1cc12109db lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.206.57@tcp remote: 0xeb28ae60c9b49292 expref: 6 pid: 157304 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4789.739539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4793.826923] Lustre: server umount lustre-OST0000 complete [ 4803.790192] Lustre: server umount lustre-MDT0000 complete [ 4804.065412] LustreError: 157309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.088527] LustreError: 157309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4806.634115] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834381 with bad export cookie 2216315398421284128 [ 4806.635657] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4806.648549] LustreError: 157288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4812.948815] Lustre: server umount lustre-MDT0001 complete [ 4824.054871] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 4826.597656] Key type lgssc unregistered [ 4826.823796] LNet: 163413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4826.831669] LNetError: 163413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4826.851455] LNet: Removed LNI 192.168.206.157@tcp [ 4827.587417] Key type .llcrypt unregistered [ 4827.589992] Key type ._llcrypt unregistered [ 4849.885694] Key type ._llcrypt registered [ 4849.887462] Key type .llcrypt registered [ 4849.986794] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 4851.074554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4851.149924] alg: No test for adler32 (adler32-zlib) [ 4852.134750] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 4852.327150] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 4853.992322] Key type lgssc registered [ 4855.727626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4864.207942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4864.238521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.959064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.288954] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4876.259184] LustreError: 164825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4876.297194] LustreError: 164825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4881.189344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4881.383065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4881.762962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4887.378031] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4896.492424] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4902.650237] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4908.625036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4908.953150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.037243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4914.755661] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4921.297730] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4930.088868] Lustre: Failing over lustre-MDT0000 [ 4930.387186] Lustre: server umount lustre-MDT0000 complete [ 4932.584589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.587691] LustreError: 166116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.599205] Lustre: Skipped 1 previous similar message [ 4932.624019] LustreError: 166116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4937.715546] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.738423] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4942.400516] LustreError: 164807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.57@tcp arrived at 1787834516 with bad export cookie 12189019881469553515 [ 4942.423105] LustreError: 164807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4942.820358] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.847029] LustreError: 164820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4947.937193] LustreError: 166116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.955065] LustreError: 166116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4948.962425] Lustre: 164138:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834507/real 1787834507] req@000000006ddc596a x1874680271105024/t0(0) o400->MGC192.168.206.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787834523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4948.993424] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.239740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.210219] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.231170] LustreError: 164822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4959.485029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4959.555655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4963.229987] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 4964.841267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4964.860135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4965.007810] Lustre: lustre-MDT0000: Denying connection for new client b9791b28-3c0c-4f4a-92cb-7ecc125958fc (at 192.168.206.57@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 4973.539105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4973.545106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.558634] Lustre: Skipped 1 previous similar message [ 4973.564424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4975.079511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4975.082547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4980.194157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.208419] Lustre: Skipped 2 previous similar messages [ 4985.328387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4985.333428] Lustre: Skipped 1 previous similar message [ 4987.360212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4987.474607] Lustre: server umount lustre-OST0000 complete [ 4990.674533] LustreError: 168558:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4990.679876] Lustre: 167907:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4990.687223] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4990.698706] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4990.708518] LustreError: 167907:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b9dd6bc7 x1874680271126016/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4990.725912] LustreError: 167907:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4990.738475] LustreError: 167907:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4990.748412] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 4990.965593] Lustre: server umount lustre-MDT0000 complete [ 4993.972432] LustreError: 164806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834568 with bad export cookie 12189019881469553620 [ 4993.999246] LustreError: 164806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4994.001698] LustreError: 164826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.006038] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4996.416739] Lustre: server umount lustre-MDT0001 complete [ 5006.919878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5009.623116] Key type lgssc unregistered [ 5009.917770] LNet: 169314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5009.931882] LNetError: 169314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5009.955074] LNet: Removed LNI 192.168.206.157@tcp [ 5010.903451] Key type .llcrypt unregistered [ 5010.905534] Key type ._llcrypt unregistered [ 5032.792769] Key type ._llcrypt registered [ 5032.794689] Key type .llcrypt registered [ 5032.876569] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5033.583784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5033.605207] alg: No test for adler32 (adler32-zlib) [ 5034.796939] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5035.042926] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5036.688152] Key type lgssc registered [ 5037.564363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5044.980646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5045.004911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.472328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.990663] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5055.979125] LustreError: 170719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.993772] LustreError: 170719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5059.185463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.498956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5063.615647] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5071.805422] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5078.545547] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5085.895821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.313177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5091.439516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5092.765729] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5100.638355] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5109.217846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.223289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.244109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5109.256637] Lustre: Skipped 1 previous similar message [ 5110.073425] Lustre: server umount lustre-OST0000 complete [ 5114.338996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.339939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.354054] Lustre: Skipped 2 previous similar messages [ 5114.362714] Lustre: Skipped 1 previous similar message [ 5119.459099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.470253] Lustre: Skipped 1 previous similar message [ 5119.984562] Lustre: server umount lustre-MDT0000 complete [ 5123.386349] LustreError: 170699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834697 with bad export cookie 10021153248639384100 [ 5123.401898] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5123.409642] LustreError: 170699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5123.681976] Lustre: server umount lustre-MDT0001 complete [ 5133.326821] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5136.117903] Key type lgssc unregistered [ 5136.413588] LNet: 174255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5136.421590] LNetError: 174255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5136.440739] LNet: Removed LNI 192.168.206.157@tcp [ 5137.132506] Key type .llcrypt unregistered [ 5137.135071] Key type ._llcrypt unregistered [ 5168.060992] Key type ._llcrypt registered [ 5168.062683] Key type .llcrypt registered [ 5168.145923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5169.244110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5169.327821] alg: No test for adler32 (adler32-zlib) [ 5170.410166] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5170.568940] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5172.208165] Key type lgssc registered [ 5173.455884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.768119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5181.796221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.476519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5188.630733] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5192.675058] LustreError: 175957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5192.680453] LustreError: 175957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5197.798855] LustreError: 175958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5198.345397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.688568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5203.382323] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5211.298034] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5217.119578] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5224.024883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5224.426716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.541070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5230.954701] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5239.422732] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5247.970971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5247.976496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5247.993510] LustreError: Skipped 1 previous similar message [ 5247.994197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5248.034076] Lustre: Skipped 1 previous similar message [ 5249.314592] Lustre: server umount lustre-OST0000 complete [ 5255.138549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5255.146713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.154212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5255.157964] Lustre: Skipped 1 previous similar message [ 5256.161419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5256.164304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5259.604457] Lustre: server umount lustre-MDT0000 complete [ 5261.282637] LustreError: 175958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.295941] LustreError: 175958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5263.186793] LustreError: 175939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834837 with bad export cookie 6914113532354131880 [ 5263.199852] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.203506] LustreError: 175939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5263.625621] Lustre: server umount lustre-MDT0001 complete [ 5274.418985] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5276.940732] Key type lgssc unregistered [ 5277.216218] LNet: 179494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5277.235965] LNetError: 179494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5277.252778] LNet: Removed LNI 192.168.206.157@tcp [ 5277.910940] Key type .llcrypt unregistered [ 5277.914361] Key type ._llcrypt unregistered [ 5296.231727] Key type ._llcrypt registered [ 5296.235933] Key type .llcrypt registered [ 5296.311715] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5297.022839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5297.039850] alg: No test for adler32 (adler32-zlib) [ 5298.111746] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5298.309925] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5300.000187] Key type lgssc registered [ 5301.130771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5310.698428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5310.715775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.991805] Lustre: *** cfs_fail_loc=135, val=0*** [ 5312.056951] LustreError: 180996:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5312.063812] LustreError: 180996:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.157@tcp: cfg command failed: rc = -2 [ 5312.081071] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5312.081071] [ 5312.087546] LustreError: MGC192.168.206.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5312.100666] LustreError: 180957:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5312.106881] LustreError: 180957:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5312.115826] LustreError: 180957:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5312.126163] LustreError: 180957:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5312.190877] Lustre: server umount lustre-MDT0000 complete [ 5312.195587] LustreError: 180957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5318.066545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.386772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.029830] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5327.843519] LustreError: 181367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.856950] LustreError: 181367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5328.903594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.210410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5332.934670] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5338.996655] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5343.793764] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5347.811132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5347.832083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5347.840147] Lustre: Skipped 1 previous similar message [ 5351.686899] Lustre: server umount lustre-MDT0000 complete [ 5352.929869] LustreError: 182095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5352.945373] LustreError: 182095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5353.956102] LustreError: 181368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.983467] LustreError: 181368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5354.707548] LustreError: 181349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834929 with bad export cookie 1788488807558669189 [ 5354.708516] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5354.718727] LustreError: 181349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5355.047360] Lustre: server umount lustre-MDT0001 complete [ 5363.328498] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5365.471894] Key type lgssc unregistered [ 5365.684299] LNet: 183690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5365.702568] LNetError: 183690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5365.715733] LNet: Removed LNI 192.168.206.157@tcp [ 5366.365796] Key type .llcrypt unregistered [ 5366.368907] Key type ._llcrypt unregistered [ 5383.775938] Key type ._llcrypt registered [ 5383.777910] Key type .llcrypt registered [ 5383.858768] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5384.068179] Key type .llcrypt unregistered [ 5384.071288] Key type ._llcrypt unregistered [ 5397.026350] Key type ._llcrypt registered [ 5397.028755] Key type .llcrypt registered [ 5397.114216] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5397.839563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5397.891396] alg: No test for adler32 (adler32-zlib) [ 5398.880467] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5399.086369] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5400.720677] Key type lgssc registered [ 5401.697068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.942989] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.956604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5424.546544] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5429.664333] LustreError: 185805:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.157@tcp: failed processing log, type 4: rc = -110 [ 5456.416836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.530578] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5469.615134] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5476.022083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.533813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5476.592329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5481.345400] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5481.956400] LustreError: 185831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.975692] LustreError: 185831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5487.083167] LustreError: 185830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.813635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5490.153084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5494.707203] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5502.222096] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5507.900175] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5510.605268] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5520.878408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5520.890504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.914782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.895513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.897170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5526.466503] Lustre: server umount lustre-OST0000 complete [ 5531.105664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5531.117737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.143174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5531.149074] Lustre: Skipped 1 previous similar message [ 5536.465449] Lustre: server umount lustre-MDT0000 complete [ 5537.257994] LustreError: 186849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.280860] LustreError: 186849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5540.606982] LustreError: 188467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835115 with bad export cookie 1944648167283180498 [ 5540.615934] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.629307] LustreError: 188467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5541.013757] Lustre: server umount lustre-MDT0001 complete [ 5551.722394] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5555.125992] Key type lgssc unregistered [ 5555.555765] LNet: 189773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5555.576877] LNetError: 189773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5556.647363] LNet: Removed LNI 192.168.206.157@tcp [ 5557.909444] Key type .llcrypt unregistered [ 5557.915270] Key type ._llcrypt unregistered [ 5585.511549] Key type ._llcrypt registered [ 5585.516773] Key type .llcrypt registered [ 5585.630455] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5587.736713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.824109] alg: No test for adler32 (adler32-zlib) [ 5589.149340] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5589.425470] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5591.128183] Key type lgssc registered [ 5592.655674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5601.908505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5601.959293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.438452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.272402] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5612.515065] LustreError: 191186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.532205] LustreError: 191186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5616.651569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.132963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5621.959429] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5628.635610] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5633.893992] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5639.985388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.279432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5646.660182] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5648.503381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5653.781419] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5658.631451] Lustre: Failing over lustre-MDT0000 [ 5658.912344] Lustre: server umount lustre-MDT0000 complete [ 5661.425476] LustreError: 193032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.179031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.191371] Lustre: Skipped 2 previous similar messages [ 5666.519639] LustreError: 191183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5666.541062] LustreError: 191183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5671.636190] LustreError: 191183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5671.657438] LustreError: 191183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5676.611827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.735090] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5676.760289] LustreError: 193032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.777031] LustreError: 193032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5676.944869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5676.997247] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5680.534668] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5681.878182] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5682.155626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5682.170171] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5682.203077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5684.367872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5693.318279] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5702.627787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.633740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5702.646636] Lustre: Skipped 1 previous similar message [ 5702.661404] Lustre: Skipped 1 previous similar message [ 5707.752441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.762031] Lustre: Skipped 1 previous similar message [ 5708.724667] Lustre: server umount lustre-OST0000 complete [ 5712.865665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.868972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5712.869604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5712.888429] Lustre: Skipped 1 previous similar message [ 5717.987383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5717.997044] Lustre: Skipped 2 previous similar messages [ 5718.278852] Lustre: server umount lustre-MDT0000 complete [ 5722.317431] LustreError: 191168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835296 with bad export cookie 1094926397094205883 [ 5722.332511] LustreError: 191168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5722.341897] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.653603] Lustre: server umount lustre-MDT0001 complete [ 5733.370356] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5736.048693] Key type lgssc unregistered [ 5736.279431] LNet: 196128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5736.294839] LNetError: 196128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5737.320704] LNet: Removed LNI 192.168.206.157@tcp [ 5738.263674] Key type .llcrypt unregistered [ 5738.270243] Key type ._llcrypt unregistered [ 5757.722842] Key type ._llcrypt registered [ 5757.726059] Key type .llcrypt registered [ 5757.816412] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5758.705509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5758.760254] alg: No test for adler32 (adler32-zlib) [ 5759.752739] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5759.933567] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5761.625521] Key type lgssc registered [ 5762.624614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5770.180716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5770.200808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.584959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.592795] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5781.996111] LustreError: 197518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5782.030345] LustreError: 197518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5787.112797] LustreError: 197519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5788.142427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.739593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.541765] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5804.437269] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5809.224308] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5814.954268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5815.344618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.417919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5820.471560] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5827.819427] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5830.661189] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5837.547726] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5852.342957] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5858.273033] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.287639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.305547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.297190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.300952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.306900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.325205] Lustre: Skipped 1 previous similar message [ 5864.136569] Lustre: server umount lustre-OST0000 complete [ 5869.539169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.544570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.556058] Lustre: Skipped 1 previous similar message [ 5869.568184] Lustre: Skipped 1 previous similar message [ 5873.949879] Lustre: server umount lustre-MDT0000 complete [ 5874.662280] LustreError: 199486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.674108] LustreError: 199486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5877.441536] LustreError: 200112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835451 with bad export cookie 12055185331999953989 [ 5877.449120] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5877.458064] LustreError: 200112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5877.780931] Lustre: server umount lustre-MDT0001 complete [ 5887.251973] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 5889.469477] Key type lgssc unregistered [ 5889.688136] LNet: 201372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5889.697652] LNetError: 201372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5889.733657] LNet: Removed LNI 192.168.206.157@tcp [ 5890.447330] Key type .llcrypt unregistered [ 5890.450497] Key type ._llcrypt unregistered [ 5906.627717] Key type ._llcrypt registered [ 5906.629392] Key type .llcrypt registered [ 5906.733878] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 5907.837446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.862594] alg: No test for adler32 (adler32-zlib) [ 5908.925937] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 5909.187221] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 5910.848196] Key type lgssc registered [ 5911.818985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5917.955331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5917.966221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.265466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.706726] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5929.443462] LustreError: 202616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.457147] LustreError: 202616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5929.737576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.061740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5933.289870] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5939.681393] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5945.060884] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5950.899493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5951.201789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.669923] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 5963.511097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5963.601997] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5969.014600] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5981.040256] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5991.188849] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6000.997277] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6009.833849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.852206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.854887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.893048] Lustre: Skipped 1 previous similar message [ 6014.953477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6014.969539] Lustre: Skipped 1 previous similar message [ 6015.906546] Lustre: server umount lustre-OST0000 complete [ 6020.069208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6020.074876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6020.086656] Lustre: Skipped 2 previous similar messages [ 6020.106440] Lustre: Skipped 1 previous similar message [ 6025.185626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6025.193837] Lustre: Skipped 1 previous similar message [ 6026.109954] Lustre: server umount lustre-MDT0000 complete [ 6029.938745] LustreError: 202597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835604 with bad export cookie 3421063457210081659 [ 6029.941856] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.953060] LustreError: 202597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6030.342788] Lustre: server umount lustre-MDT0001 complete [ 6041.109739] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6043.823679] Key type lgssc unregistered [ 6044.182849] LNet: 207689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6044.195196] LNetError: 207689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6044.211074] LNet: Removed LNI 192.168.206.157@tcp [ 6045.130442] Key type .llcrypt unregistered [ 6045.133896] Key type ._llcrypt unregistered [ 6070.340944] Key type ._llcrypt registered [ 6070.344525] Key type .llcrypt registered [ 6070.499973] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6071.624746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6071.730389] alg: No test for adler32 (adler32-zlib) [ 6072.891846] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6073.128936] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6074.808238] Key type lgssc registered [ 6076.094898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6084.615711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6084.633695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.231451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6091.312376] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6096.370097] LustreError: 209103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.423165] LustreError: 209103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6100.635731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.881037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6105.601319] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6112.970487] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6119.388035] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6126.856520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6127.355598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.472434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6133.582861] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6142.172949] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6161.380485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6161.393172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6161.398018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6161.418622] Lustre: Skipped 1 previous similar message [ 6166.417915] Lustre: server umount lustre-OST0000 complete [ 6171.619458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.631463] Lustre: Skipped 1 previous similar message [ 6171.639183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6171.641311] Lustre: Skipped 1 previous similar message [ 6175.930267] Lustre: server umount lustre-MDT0000 complete [ 6176.738098] LustreError: 209099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.763160] LustreError: 209099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6178.930111] LustreError: 209084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835753 with bad export cookie 3518739605634985136 [ 6178.936431] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.942410] LustreError: 209084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6179.375239] Lustre: server umount lustre-MDT0001 complete [ 6188.676428] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6190.996619] Key type lgssc unregistered [ 6191.259651] LNet: 212803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6191.271879] LNetError: 212803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6191.299894] LNet: Removed LNI 192.168.206.157@tcp [ 6192.062719] Key type .llcrypt unregistered [ 6192.067312] Key type ._llcrypt unregistered [ 6210.597433] Key type ._llcrypt registered [ 6210.600272] Key type .llcrypt registered [ 6210.726038] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6211.973716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6212.267311] alg: No test for adler32 (adler32-zlib) [ 6213.684631] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6213.975819] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6215.779048] Key type lgssc registered [ 6217.761843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6227.926062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6227.955504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6229.724202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6234.183697] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6240.227959] LustreError: 214073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.245979] LustreError: 214073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6242.138626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6242.469598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6246.390947] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6253.774197] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6258.976291] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6266.951946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.478189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.574442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6275.722255] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6284.823766] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.137144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6300.137788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.147160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6300.178963] Lustre: Skipped 1 previous similar message [ 6305.265705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6305.274345] Lustre: Skipped 1 previous similar message [ 6305.754495] Lustre: server umount lustre-OST0000 complete [ 6310.371162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.379309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6310.393810] Lustre: Skipped 1 previous similar message [ 6310.415585] Lustre: Skipped 1 previous similar message [ 6315.491103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6315.495533] Lustre: Skipped 1 previous similar message [ 6315.816288] Lustre: server umount lustre-MDT0000 complete [ 6319.554058] LustreError: 214052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835894 with bad export cookie 1255460816877772717 [ 6319.569705] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6319.575946] LustreError: 214052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6319.882519] Lustre: server umount lustre-MDT0001 complete [ 6329.875804] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6332.519573] Key type lgssc unregistered [ 6332.769205] LNet: 217758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6332.780384] LNetError: 217758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6332.791211] LNet: Removed LNI 192.168.206.157@tcp [ 6333.582794] Key type .llcrypt unregistered [ 6333.584743] Key type ._llcrypt unregistered [ 6351.350655] Key type ._llcrypt registered [ 6351.353858] Key type .llcrypt registered [ 6351.453362] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6352.345120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6352.476060] alg: No test for adler32 (adler32-zlib) [ 6353.516868] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6353.726437] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6355.392152] Key type lgssc registered [ 6356.700827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6364.394568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.415545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6365.866042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6370.292738] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6376.418545] LustreError: 219025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.435435] LustreError: 219025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6378.023051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.282749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6382.429495] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6390.335426] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6397.535384] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.246516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6407.084556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6411.224327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6414.724317] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6423.305715] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6433.863157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.037212] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6434.342623] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6441.433740] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6449.559520] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6451.274313] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6462.858675] Lustre: Permanently deactivating lustre-OST0001 [ 6462.871408] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6472.801787] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6482.278651] Lustre: Permanently reactivating lustre-OST0001 [ 6482.294842] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6482.306508] Lustre: Skipped 2 previous similar messages [ 6482.611383] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.631368] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6482.638020] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6482.663320] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6482.690261] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6495.220739] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.229236] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6495.239490] Lustre: Skipped 2 previous similar messages [ 6495.254707] Lustre: Skipped 1 previous similar message [ 6500.333089] LustreError: 220845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.357612] LustreError: 220845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6500.416630] Lustre: server umount lustre-OST0001 complete [ 6505.442509] LustreError: 220845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.446670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.448822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.450269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.477681] LustreError: 220845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6505.525779] Lustre: Skipped 1 previous similar message [ 6510.553995] Lustre: server umount lustre-OST0000 complete [ 6515.690847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.692302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.700158] Lustre: Skipped 1 previous similar message [ 6515.705698] Lustre: Skipped 1 previous similar message [ 6520.480405] Lustre: server umount lustre-MDT0000 complete [ 6520.803990] LustreError: 219024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.822515] LustreError: 219024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6523.726168] LustreError: 219006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836098 with bad export cookie 7610467949542722995 [ 6523.728329] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6523.734055] LustreError: 219006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6523.991694] Lustre: server umount lustre-MDT0001 complete [ 6532.441173] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6535.411680] Key type lgssc unregistered [ 6535.630771] LNet: 224126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6535.636319] LNetError: 224126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6535.659491] LNet: Removed LNI 192.168.206.157@tcp [ 6536.655231] Key type .llcrypt unregistered [ 6536.657441] Key type ._llcrypt unregistered [ 6579.230301] Key type ._llcrypt registered [ 6579.233129] Key type .llcrypt registered [ 6579.351753] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6580.863750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6580.910311] alg: No test for adler32 (adler32-zlib) [ 6582.236793] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6582.605928] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6584.296155] Key type lgssc registered [ 6586.420171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6594.484972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6594.496305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6595.697887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6595.725145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6595.918881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.992215] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6610.643847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.784626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6610.845770] Lustre: 226333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6611.037705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6611.258611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6616.443093] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6626.231926] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6633.490228] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6641.256841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6641.464669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6641.737712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.530441] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6651.065580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6656.443147] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.298467] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6668.050886] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6673.616468] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6681.152836] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6697.318168] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6697.331952] Lustre: Skipped 1 previous similar message [ 6717.887495] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6717.893055] Lustre: Skipped 2 previous similar messages [ 6753.920262] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6753.924265] Lustre: Skipped 4 previous similar messages [ 6819.522144] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6819.544376] Lustre: Skipped 7 previous similar messages [ 6830.054681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6830.067708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.097482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6835.037651] Lustre: server umount lustre-OST0000 complete [ 6840.801857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.812160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6840.818621] Lustre: Skipped 2 previous similar messages [ 6840.844573] Lustre: Skipped 2 previous similar messages [ 6845.532962] Lustre: server umount lustre-MDT0000 complete [ 6851.042167] LustreError: 226339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6851.081180] LustreError: 226339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6851.331130] LustreError: 226323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836425 with bad export cookie 18294329739924002067 [ 6851.332750] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6851.351900] LustreError: 226323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6851.912121] Lustre: server umount lustre-MDT0001 complete [ 6863.460359] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 6866.736327] Key type lgssc unregistered [ 6867.065533] LNet: 231136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6867.070925] LNetError: 231136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6867.103716] LNet: Removed LNI 192.168.206.157@tcp [ 6867.989374] Key type .llcrypt unregistered [ 6867.993125] Key type ._llcrypt unregistered [ 6892.544820] Key type ._llcrypt registered [ 6892.548402] Key type .llcrypt registered [ 6892.753996] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 6894.083839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6894.112872] alg: No test for adler32 (adler32-zlib) [ 6895.275246] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 6895.528544] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 6897.210098] Key type lgssc registered [ 6898.388599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6906.522989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6906.543673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.188279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.223249] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6917.634370] LustreError: 232551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.655690] LustreError: 232551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6922.331366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6922.801947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6927.671826] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6935.242882] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6941.417347] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6948.378073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.703795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6952.837244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6955.517710] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_default_debug -1 all [ 6962.759629] Lustre: DEBUG MARKER: oleg657-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6983.649090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6983.651252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.654078] LustreError: Skipped 1 previous similar message [ 6983.654729] LustreError: 234375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.668752] Lustre: Skipped 1 previous similar message [ 6983.746764] Lustre: server umount lustre-OST0000 complete [ 6988.770520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.803160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.810133] Lustre: Skipped 1 previous similar message [ 6993.374914] Lustre: server umount lustre-MDT0000 complete [ 6993.892350] LustreError: 232547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.915443] LustreError: 232547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6996.574824] LustreError: 234395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836571 with bad export cookie 10256281578128860982 [ 6996.585586] LustreError: MGC192.168.206.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.586750] LustreError: 234395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6996.981081] Lustre: server umount lustre-MDT0001 complete [ 7006.909704] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7009.321376] Key type lgssc unregistered [ 7009.600369] LNet: 236244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7009.605949] LNetError: 236244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7009.623263] LNet: Removed LNI 192.168.206.157@tcp [ 7010.480375] Key type .llcrypt unregistered [ 7010.482250] Key type ._llcrypt unregistered [ 7031.354470] Key type ._llcrypt registered [ 7031.362222] Key type .llcrypt registered [ 7031.482309] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7032.600197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7032.716386] alg: No test for adler32 (adler32-zlib) [ 7033.789853] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7034.039673] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7035.712265] Key type lgssc registered [ 7036.768331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7309.442424] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7312.728751] Key type lgssc unregistered [ 7313.110789] LNet: 238282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7313.118163] LNetError: 238282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7313.144937] LNet: Removed LNI 192.168.206.157@tcp [ 7314.160696] Key type .llcrypt unregistered [ 7314.163494] Key type ._llcrypt unregistered [ 7344.246854] Key type ._llcrypt registered [ 7344.248975] Key type .llcrypt registered [ 7344.365919] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing set_hostid [ 7360.947642] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing load_modules_local [ 7362.123979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.209514] alg: No test for adler32 (adler32-zlib) [ 7363.279209] Lustre: Lustre: Build Version: 2.17.57_64_gb96373c [ 7363.539518] LNet: Added LNI 192.168.206.157@tcp [8/256/0/180] [ 7365.208931] Key type lgssc registered [ 7366.586968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7399.177966] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7082 sec ========== 09:22:53 (1787836973) [ 7400.593520] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:22:55 (1787836975) === [ 7404.386932] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:22:58 (1787836978) === [ 7447.406188] Lustre: DEBUG MARKER: oleg657-server.virtnet: executing unload_modules_local [ 7450.682638] Key type lgssc unregistered [ 7451.100609] LNet: 243522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7451.121129] LNetError: 243522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7451.144475] LNet: Removed LNI 192.168.206.157@tcp [ 7452.114509] Key type .llcrypt unregistered [ 7452.119605] Key type ._llcrypt unregistered