[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456344354 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003288] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012052] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.021629] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025609] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030052] *** VALIDATE tmpfs *** [ 0.031463] *** VALIDATE proc *** [ 0.033109] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043221] debug: unmapping init [mem 0xffffffffa7859000-0xffffffffa7860fff] [ 0.045188] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046643] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.054277] rcu: Hierarchical SRCU implementation. [ 0.056424] smp: Bringing up secondary CPUs ... [ 0.057518] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061644] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.168102] node 0 deferred pages initialised in 102ms [ 0.171250] devtmpfs: initialized [ 0.172227] x86/mm: Memory block size: 128MB [ 0.175711] gcov: version magic: 0x41383552 [ 0.178075] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.181106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183353] pinctrl core: initialized pinctrl subsystem [ 0.185196] [ 0.185749] ************************************************************* [ 0.189016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190009] ** ** [ 0.192010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.194009] ** ** [ 0.196011] ** This means that this kernel is built to expose internal ** [ 0.198013] ** IOMMU data structures, which may compromise security on ** [ 0.200013] ** your system. ** [ 0.202016] ** ** [ 0.204013] ** If you see this message and you are not debugging the ** [ 0.206013] ** kernel, report this immediately to your vendor! ** [ 0.208015] ** ** [ 0.210011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212026] ************************************************************* [ 0.215183] NET: Registered protocol family 16 [ 0.216439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.218075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.221062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.225156] cpuidle: using governor menu [ 0.226832] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227448] PCI: Using configuration type 1 for base access [ 0.228170] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.238086] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.242056] cryptd: max_cpu_qlen set to 1000 [ 0.244228] ACPI: Added _OSI(Module Device) [ 0.246016] ACPI: Added _OSI(Processor Device) [ 0.248015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.254844] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261371] ACPI: Interpreter enabled [ 0.262063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264015] ACPI: Using IOAPIC for interrupt routing [ 0.266098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.271400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281634] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.289077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.294263] acpiphp: Slot [2] registered [ 0.296127] acpiphp: Slot [5] registered [ 0.297148] acpiphp: Slot [6] registered [ 0.299109] acpiphp: Slot [7] registered [ 0.301106] acpiphp: Slot [8] registered [ 0.302105] acpiphp: Slot [9] registered [ 0.303079] acpiphp: Slot [10] registered [ 0.304077] acpiphp: Slot [3] registered [ 0.305047] acpiphp: Slot [4] registered [ 0.306105] acpiphp: Slot [11] registered [ 0.308092] acpiphp: Slot [12] registered [ 0.309090] acpiphp: Slot [13] registered [ 0.311099] acpiphp: Slot [14] registered [ 0.312108] acpiphp: Slot [15] registered [ 0.314125] acpiphp: Slot [16] registered [ 0.315105] acpiphp: Slot [17] registered [ 0.316079] acpiphp: Slot [18] registered [ 0.317092] acpiphp: Slot [19] registered [ 0.319070] acpiphp: Slot [20] registered [ 0.321176] acpiphp: Slot [21] registered [ 0.322106] acpiphp: Slot [22] registered [ 0.324099] acpiphp: Slot [23] registered [ 0.325091] acpiphp: Slot [24] registered [ 0.326077] acpiphp: Slot [25] registered [ 0.328085] acpiphp: Slot [26] registered [ 0.329075] acpiphp: Slot [27] registered [ 0.330100] acpiphp: Slot [28] registered [ 0.332122] acpiphp: Slot [29] registered [ 0.333073] acpiphp: Slot [30] registered [ 0.334000] acpiphp: Slot [31] registered [ 0.336090] PCI host bridge to bus 0000:00 [ 0.338023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.348035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357056] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375484] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384601] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388102] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394913] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.400018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.413023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.419867] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.426799] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.434020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.466591] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.471016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.479016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.512651] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.518017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.536021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.547077] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.553016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.558017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583130] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.589014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.608018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.617870] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.632016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.659987] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.662397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665633] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.668384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.671216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676049] iommu: Default domain type: Passthrough [ 0.677356] SCSI subsystem initialized [ 0.679139] ACPI: bus type USB registered [ 0.680089] usbcore: registered new interface driver usbfs [ 0.682087] usbcore: registered new interface driver hub [ 0.683060] usbcore: registered new device driver usb [ 0.685197] pps_core: LinuxPPS API ver. 1 registered [ 0.686008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689060] PTP clock support registered [ 0.691052] EDAC MC: Ver: 3.0.0 [ 0.693191] PCI: Using ACPI for IRQ routing [ 0.695094] NetLabel: Initializing [ 0.696013] NetLabel: domain hash size = 128 [ 0.697015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.698122] NetLabel: unlabeled traffic allowed by default [ 0.701139] vgaarb: loaded [ 0.702283] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.705016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714342] clocksource: Switched to clocksource kvm-clock [ 0.826347] VFS: Disk quotas dquot_6.6.0 [ 0.828242] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.831455] *** VALIDATE ramfs *** [ 0.832843] *** VALIDATE hugetlbfs *** [ 0.834559] pnp: PnP ACPI init [ 0.837028] pnp: PnP ACPI: found 6 devices [ 0.857706] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.862090] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.864617] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.867265] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.870241] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.873084] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.876296] NET: Registered protocol family 2 [ 0.878927] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.883963] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.887349] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.893135] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.897395] TCP: Hash tables configured (established 65536 bind 65536) [ 0.900702] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.904409] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907901] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911429] NET: Registered protocol family 1 [ 0.915066] RPC: Registered named UNIX socket transport module. [ 0.917779] RPC: Registered udp transport module. [ 0.919897] RPC: Registered tcp transport module. [ 0.921954] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924521] NET: Registered protocol family 44 [ 0.926470] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.929014] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931516] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.934531] PCI: CLS 0 bytes, default 64 [ 0.936705] Unpacking initramfs... [ 2.376486] debug: unmapping init [mem 0xffff9452fcc54000-0xffff9452fffbffff] [ 2.384458] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.387415] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.392079] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.883340] Initialise system trusted keyrings [ 2.884905] Key type blacklist registered [ 2.886507] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.898145] zbud: loaded [ 2.901644] *** VALIDATE nfs *** [ 2.903171] *** VALIDATE nfs4 *** [ 2.905142] pstore: using deflate compression [ 2.909586] Platform Keyring initialized [ 3.015725] NET: Registered protocol family 38 [ 3.018218] Key type asymmetric registered [ 3.020189] Asymmetric key parser 'x509' registered [ 3.022560] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.026963] io scheduler mq-deadline registered [ 3.028809] io scheduler kyber registered [ 3.030482] io scheduler bfq registered [ 3.032458] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.034989] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.039451] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.043708] ACPI: Power Button [PWRF] [ 3.049659] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.055958] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.070773] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.078437] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.094540] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.122592] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152528] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158386] Non-volatile memory driver v1.3 [ 3.160436] Linux agpgart interface v0.103 [ 3.196193] virtio_blk virtio1: [vda] 146976 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.199620] vda: detected capacity change from 0 to 75251712 [ 3.214682] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.217676] vdb: detected capacity change from 0 to 1073741824 [ 3.232921] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.235723] vdc: detected capacity change from 0 to 2621440000 [ 3.249611] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.252809] vdd: detected capacity change from 0 to 2621440000 [ 3.266878] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.270163] vde: detected capacity change from 0 to 4294967296 [ 3.284127] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.286462] vdf: detected capacity change from 0 to 4294967296 [ 3.292996] libphy: Fixed MDIO Bus: probed [ 3.298965] usbcore: registered new interface driver usbserial_generic [ 3.302097] usbserial: USB Serial support registered for generic [ 3.304893] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.308695] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.310428] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.312556] mousedev: PS/2 mouse device common for all mice [ 3.315016] rtc_cmos 00:05: RTC can wake from S4 [ 3.317971] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.319386] rtc_cmos 00:05: registered as rtc0 [ 3.322739] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.323724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.326297] intel_pstate: CPU model not supported [ 3.330290] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.337243] hid: raw HID events driver (C) Jiri Kosina [ 3.339388] usbcore: registered new interface driver usbhid [ 3.341679] usbhid: USB HID core driver [ 3.342901] drop_monitor: Initializing network drop monitor service [ 3.347049] Initializing XFRM netlink socket [ 3.349677] NET: Registered protocol family 10 [ 3.353335] Segment Routing with IPv6 [ 3.354622] NET: Registered protocol family 17 [ 3.356464] mpls_gso: MPLS GSO support [ 3.361451] RAS: Correctable Errors collector initialized. [ 3.363862] AVX version of gcm_enc/dec engaged. [ 3.365957] AES CTR mode by8 optimization enabled [ 3.446479] sched_clock: Marking stable (3446450242, 0)->(4343890516, -897440274) [ 3.449913] registered taskstats version 1 [ 3.451679] Loading compiled-in X.509 certificates [ 3.454097] zswap: loaded using pool lzo/zbud [ 3.479619] Key type big_key registered [ 3.493606] Key type encrypted registered [ 3.495591] ima: No TPM chip found, activating TPM-bypass! [ 3.497299] ima: Allocated hash algorithm: sha1 [ 3.498791] ima: No architecture policies found [ 3.500416] evm: Initialising EVM extended attributes: [ 3.501940] evm: security.selinux [ 3.503194] evm: security.ima [ 3.504078] evm: security.capability [ 3.505208] evm: HMAC attrs: 0x1 [ 3.507525] rtc_cmos 00:05: setting system clock to 2026-08-27 21:01:24 UTC (1787864484) [ 3.513647] debug: unmapping init [mem 0xffffffffa8803000-0xffffffffa89fffff] [ 3.516304] debug: unmapping init [mem 0xffffffffa7582000-0xffffffffa7858fff] [ 3.524098] Write protecting the kernel read-only data: 28672k [ 3.527357] debug: unmapping init [mem 0xffffffffa5c03000-0xffffffffa5dfffff] [ 3.530279] debug: unmapping init [mem 0xffffffffa6514000-0xffffffffa65fffff] [ 3.563241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.570947] systemd[1]: Detected virtualization kvm. [ 3.572667] systemd[1]: Detected architecture x86-64. [ 3.574265] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.599569] systemd[1]: No hostname configured. [ 3.602155] systemd[1]: Set hostname to . [ 3.606764] random: systemd: uninitialized urandom read (16 bytes read) [ 3.609700] systemd[1]: Initializing machine ID from random generator. [ 3.752730] random: systemd: uninitialized urandom read (16 bytes read) [ 3.756215] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.760543] random: systemd: uninitialized urandom read (16 bytes read) [ 3.764403] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.770297] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.471350] device-mapper: uevent: version 1.0.3 [ 4.473360] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.220829] random: fast init done [ 5.257526] virtio_net virtio0 ens2: renamed from eth0 [ 5.384319] scsi host0: ata_piix [ 5.401847] scsi host1: ata_piix [ 5.403633] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.405928] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.082600] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.013408] random: crng init done [ 10.015605] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.442227] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.527794] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.749440] SELinux: Disabled at runtime. [ 11.810888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.819125] systemd[1]: Detected virtualization kvm. [ 11.820871] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.230160] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.233522] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.238363] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.244285] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.247899] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.255000] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.259607] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.412715] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.648918] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.886953] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.917995] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.988797] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.999792] EDAC sbridge: Ver: 1.1.2 [ 14.591465] Key type dns_resolver registered [ 14.892743] NFS: Registering the id_resolver key type [ 14.895286] Key type id_resolver registered [ 14.896823] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-server login: [ 46.345015] hrtimer: interrupt took 3022758 ns [ 54.712799] libcfs: loading out-of-tree module taints kernel. [ 54.929790] Key type ._llcrypt registered [ 54.932798] Key type .llcrypt registered [ 55.220579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 74.996569] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 76.430709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.444726] alg: No test for adler32 (adler32-zlib) [ 77.948512] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 78.611241] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 80.368241] Key type lgssc registered [ 82.111361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.186684] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.157746] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 152.558213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.586259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.771783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.819868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.906882] Lustre: lustre-MDT0000: new disk, initializing [ 153.984769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.007570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 157.145826] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.926590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.013181] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 168.036781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 168.041350] Lustre: Skipped 1 previous similar message [ 168.120928] Lustre: lustre-MDT0001: new disk, initializing [ 168.193085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.223219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.245728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.026477] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.988722] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.117217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 184.368486] Lustre: lustre-OST0000: new disk, initializing [ 184.374133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 184.380652] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 184.432512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.012944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.018125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.102797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.133634] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.575972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.905569] Lustre: lustre-OST0001: new disk, initializing [ 203.910692] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.919361] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.042247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 209.510521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.530350] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.639801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.219847] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.573923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.537927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.144340] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing check_logdir /tmp/testlogs/ [ 239.893912] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing yml_node [ 243.292253] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 245.758512] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 248.298406] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 250.004788] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 17:05:29 EDT 2026 [ 267.488075] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 268.782423] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 286.177789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.186561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.209737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.689814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.714352] Lustre: Skipped 1 previous similar message [ 286.724831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.734137] Lustre: Skipped 2 previous similar messages [ 290.556544] Lustre: server umount lustre-MDT0000 complete [ 293.978427] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864774 with bad export cookie 1592003182496535098 [ 293.987621] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.992584] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 296.930356] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.931517] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.932258] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.948176] Lustre: Skipped 2 previous similar messages [ 296.960979] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 298.977059] LustreError: 9529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.977714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.994475] Lustre: Skipped 2 previous similar messages [ 298.995477] LustreError: 9529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 300.379296] Lustre: server umount lustre-MDT0001 complete [ 304.362114] Lustre: server umount lustre-OST0000 complete [ 308.713886] Lustre: server umount lustre-OST0001 complete [ 315.438611] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 326.007370] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 362.442112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 362.709534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 362.731254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 362.814213] Lustre: lustre-MDT0000: new disk, initializing [ 362.956591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.982039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 367.036630] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 377.758042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.869818] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 377.950170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 377.959481] Lustre: Skipped 1 previous similar message [ 378.036743] Lustre: lustre-MDT0001: new disk, initializing [ 378.165988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.215825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.231186] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.356164] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 392.674440] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 399.041953] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 406.302819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 406.636601] Lustre: lustre-OST0000: new disk, initializing [ 406.645563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 406.663819] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 406.801276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.329843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 408.344504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 408.481194] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 412.985390] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 423.763705] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 428.991106] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 429.248473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 433.795334] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 434.039244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 439.267741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.279449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.291777] LustreError: Skipped 1 previous similar message [ 439.300080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 439.310051] Lustre: Skipped 1 previous similar message [ 442.361718] Lustre: server umount lustre-OST0000 complete [ 449.511177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.527350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 449.536888] Lustre: Skipped 2 previous similar messages [ 449.549318] Lustre: Skipped 1 previous similar message [ 452.684393] Lustre: server umount lustre-MDT0000 complete [ 454.632783] LustreError: 15087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 454.665617] LustreError: 15087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 456.437758] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864937 with bad export cookie 1592003182496536631 [ 456.444103] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.453449] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 456.879674] Lustre: server umount lustre-MDT0001 complete [ 460.684974] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:08:59 (1787864939) [ 467.470465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.942715] LustreError: 19309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.008941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.866867] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 473.062734] LustreError: 19310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.899804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.190849] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 497.368872] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 503.266346] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.349173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.428493] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 526.921741] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 536.548636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.554928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.566740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.586559] Lustre: Skipped 1 previous similar message [ 540.278515] Lustre: server umount lustre-OST0000 complete [ 550.479797] Lustre: server umount lustre-MDT0000 complete [ 551.907445] LustreError: 19309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.938403] LustreError: 19309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 555.250295] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865036 with bad export cookie 1592003182496537513 [ 555.256560] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.276712] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 555.729535] Lustre: server umount lustre-MDT0001 complete [ 569.565918] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 572.874606] Key type lgssc unregistered [ 573.203785] LNet: 22846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.211612] LNetError: 22846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.246469] LNet: Removed LNI 192.168.203.150@tcp [ 574.315297] Key type .llcrypt unregistered [ 574.319873] Key type ._llcrypt unregistered [ 599.731103] Key type ._llcrypt registered [ 599.734443] Key type .llcrypt registered [ 599.833641] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 601.112559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.161433] alg: No test for adler32 (adler32-zlib) [ 602.370971] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 602.783456] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 604.520390] Key type lgssc registered [ 606.035865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.086386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 615.118938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.677849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.264637] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 626.158768] LustreError: 24261:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.193892] LustreError: 24261:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 629.366233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.845765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 634.856287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 641.999796] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.349611] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.307765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.611510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.786521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 663.493030] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 672.244398] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 689.636269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.648147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.653744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 689.661138] Lustre: Skipped 1 previous similar message [ 690.988507] Lustre: server umount lustre-OST0000 complete [ 696.290088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 696.305595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.346053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.359689] Lustre: Skipped 1 previous similar message [ 699.874523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 699.882478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 701.291503] Lustre: server umount lustre-MDT0000 complete [ 705.004800] LustreError: 24256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.030309] LustreError: 24256:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 705.582945] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865186 with bad export cookie 3561225519930816787 [ 705.587573] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.593538] LustreError: 24244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 705.990394] Lustre: server umount lustre-MDT0001 complete [ 718.944172] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 721.828857] Key type lgssc unregistered [ 722.163529] LNet: 28242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.172582] LNetError: 28242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.238190] LNet: Removed LNI 192.168.203.150@tcp [ 724.099995] Key type .llcrypt unregistered [ 724.105589] Key type ._llcrypt unregistered [ 749.421141] Key type ._llcrypt registered [ 749.423458] Key type .llcrypt registered [ 749.583249] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 751.069640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 751.260094] alg: No test for adler32 (adler32-zlib) [ 752.384686] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 752.625098] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 754.296164] Key type lgssc registered [ 755.457928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 763.753914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 763.766508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.244843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.614461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 775.649187] LustreError: 29656:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.670150] LustreError: 29656:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 779.087164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.511295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 784.159244] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 792.287528] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 798.907538] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 813.840531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.146898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.872627] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 830.824904] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 836.676603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 844.781458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 844.794185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.828246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 846.819378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 846.822042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 846.840819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.945483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.960167] Lustre: Skipped 1 previous similar message [ 857.064584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 857.080995] Lustre: Skipped 1 previous similar message [ 858.080948] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 858.207593] Lustre: server umount lustre-OST0000 complete [ 861.665622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 861.669607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.685190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.698708] Lustre: Skipped 1 previous similar message [ 867.684558] Lustre: server umount lustre-MDT0000 complete [ 871.763659] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865352 with bad export cookie 5009553568954108020 [ 871.772111] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 871.777296] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 872.188954] Lustre: server umount lustre-MDT0001 complete [ 883.250702] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 885.941574] Key type lgssc unregistered [ 886.280646] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.290588] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.322628] LNet: Removed LNI 192.168.203.150@tcp [ 887.226984] Key type .llcrypt unregistered [ 887.229675] Key type ._llcrypt unregistered [ 910.898658] Key type ._llcrypt registered [ 910.901623] Key type .llcrypt registered [ 911.019056] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 912.459260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.493263] alg: No test for adler32 (adler32-zlib) [ 913.525509] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 913.822330] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 915.464151] Key type lgssc registered [ 916.507181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.812968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.839853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.431147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.365887] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 936.929859] LustreError: 34984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.983865] LustreError: 34984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 941.680505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.058373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 942.331454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 948.010928] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 957.630870] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 965.526744] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 972.884109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 973.441912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.068821] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 987.434743] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 996.986253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1007.588528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.589677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.630259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1007.637288] Lustre: Skipped 1 previous similar message [ 1012.707278] Lustre: server umount lustre-OST0000 complete [ 1017.825936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.838531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.839099] Lustre: Skipped 2 previous similar messages [ 1017.852183] Lustre: Skipped 1 previous similar message [ 1022.531052] Lustre: server umount lustre-MDT0000 complete [ 1022.955383] LustreError: 34983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.983572] LustreError: 34983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1026.045521] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865507 with bad export cookie 9499078526188786767 [ 1026.066468] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.430716] Lustre: server umount lustre-MDT0001 complete [ 1037.323877] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1040.554255] Key type lgssc unregistered [ 1040.925572] LNet: 38546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.944155] LNetError: 38546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.978772] LNet: Removed LNI 192.168.203.150@tcp [ 1041.984789] Key type .llcrypt unregistered [ 1041.989410] Key type ._llcrypt unregistered [ 1066.689566] Key type ._llcrypt registered [ 1066.691209] Key type .llcrypt registered [ 1066.793965] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1068.136030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.181480] alg: No test for adler32 (adler32-zlib) [ 1069.319133] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1069.599278] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1071.304971] Key type lgssc registered [ 1072.477591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1081.077307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1081.129135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.832286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.792788] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1092.141378] LustreError: 39961:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1092.181516] LustreError: 39961:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1097.191716] LustreError: 39962:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.701902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.428922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1103.751982] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1111.960754] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.908969] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1126.946366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.392501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.566177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1133.616952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1141.420333] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1152.993789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1152.998392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.018916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.685587] Lustre: lustre-OST0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1154.020731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.813389] Lustre: server umount lustre-OST0000 complete [ 1176.995934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.030726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.040498] Lustre: Skipped 3 previous similar messages [ 1180.003982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.011204] Lustre: Skipped 1 previous similar message [ 1185.251930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.259806] Lustre: Skipped 1 previous similar message [ 1191.392279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.670171] Lustre: server umount lustre-MDT0000 complete [ 1194.467296] LustreError: 39962:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1194.490161] LustreError: 39962:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1195.487515] LustreError: 39943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865676 with bad export cookie 9875816610574867476 [ 1195.501971] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1195.506068] LustreError: 39943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1201.910435] Lustre: server umount lustre-MDT0001 complete [ 1211.621677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1214.457577] Key type lgssc unregistered [ 1214.854885] LNet: 43680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.866748] LNetError: 43680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.897590] LNet: Removed LNI 192.168.203.150@tcp [ 1215.727696] Key type .llcrypt unregistered [ 1215.729460] Key type ._llcrypt unregistered [ 1241.201842] Key type ._llcrypt registered [ 1241.205350] Key type .llcrypt registered [ 1241.334411] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1242.505609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.636362] alg: No test for adler32 (adler32-zlib) [ 1243.927635] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1244.281823] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1245.985404] Key type lgssc registered [ 1247.342493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1256.524793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1256.540850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.029100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.104283] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1267.175569] LustreError: 45093:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.202217] LustreError: 45093:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1272.295237] LustreError: 45094:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.215461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.792592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.398243] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1288.636364] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1295.884194] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1303.165968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.499532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1310.768936] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1319.070093] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1327.093699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1335.974337] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1337.315089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.341799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.347923] Lustre: Skipped 1 previous similar message [ 1337.369582] Lustre: Skipped 2 previous similar messages [ 1341.031232] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.50@tcp (stopping) [ 1342.427823] Lustre: server umount lustre-MDT0000 complete [ 1342.456708] LustreError: 45821:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.153871] LustreError: 45088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.184524] LustreError: 45088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1346.759158] LustreError: 45073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865827 with bad export cookie 3091100582167216501 [ 1346.780328] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.554308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.568310] Lustre: Skipped 1 previous similar message [ 1347.578278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1351.281720] LustreError: 45088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.316900] LustreError: 45088:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1351.600746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1351.605939] Lustre: Skipped 1 previous similar message [ 1361.377878] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1361.556226] LustreError: 45090:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.591043] LustreError: 45090:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 1361.698124] Lustre: server umount lustre-MDT0001 complete [ 1377.055101] Lustre: server umount lustre-OST0000 complete [ 1396.253303] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1399.270561] Key type lgssc unregistered [ 1399.594121] LNet: 48986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.605420] LNetError: 48986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.644814] LNet: Removed LNI 192.168.203.150@tcp [ 1400.644598] Key type .llcrypt unregistered [ 1400.646445] Key type ._llcrypt unregistered [ 1423.807250] Key type ._llcrypt registered [ 1423.809970] Key type .llcrypt registered [ 1423.966392] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1425.168194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1425.208093] alg: No test for adler32 (adler32-zlib) [ 1426.291657] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1426.554518] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1428.272216] Key type lgssc registered [ 1429.569057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.590272] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.616907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.281826] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1458.400297] LustreError: 50373:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 1485.088227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.134835] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1498.938528] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1663.457717] Lustre: server umount lustre-OST0000 complete [ 1680.104765] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1683.136970] Key type lgssc unregistered [ 1683.417096] LNet: 52051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.428320] LNetError: 52051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.447344] LNet: Removed LNI 192.168.203.150@tcp [ 1684.379902] Key type .llcrypt unregistered [ 1684.384480] Key type ._llcrypt unregistered [ 1708.325554] Key type ._llcrypt registered [ 1708.327632] Key type .llcrypt registered [ 1708.472862] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1709.882233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.968413] alg: No test for adler32 (adler32-zlib) [ 1711.110783] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1711.446793] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1713.192539] Key type lgssc registered [ 1714.386071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.247182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.280878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.828361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.288898] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1733.090660] LustreError: 53469:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.116281] LustreError: 53469:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1737.624646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.167981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.107660] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1751.001433] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1757.205140] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1764.436162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.103608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.206986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1771.669514] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1779.582907] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1783.778493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.798503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.808125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1784.871107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.150484] Lustre: server umount lustre-OST0000 complete [ 1794.529635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.542781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.579930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.590657] Lustre: Skipped 1 previous similar message [ 1798.734168] Lustre: server umount lustre-MDT0000 complete [ 1800.161637] LustreError: 53465:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1800.195847] LustreError: 53465:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1802.924189] LustreError: 53448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866283 with bad export cookie 9346846558753784936 [ 1802.935755] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.944990] LustreError: 53448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1803.342909] Lustre: server umount lustre-MDT0001 complete [ 1814.998357] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 1818.239427] Key type lgssc unregistered [ 1818.530737] LNet: 57002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.540700] LNetError: 57002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.570886] LNet: Removed LNI 192.168.203.150@tcp [ 1819.658162] Key type .llcrypt unregistered [ 1819.663331] Key type ._llcrypt unregistered [ 1843.615191] Key type ._llcrypt registered [ 1843.617929] Key type .llcrypt registered [ 1843.736751] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 1844.652213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.687032] alg: No test for adler32 (adler32-zlib) [ 1845.767914] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 1846.002487] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 1847.768516] Key type lgssc registered [ 1849.037229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.153817] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1857.185070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1872.736371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1877.856283] LustreError: 58386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 1904.608409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.445880] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1918.835385] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1925.187988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.706157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.769870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1929.418325] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1930.728426] LustreError: 58412:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.762742] LustreError: 58412:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1935.842159] LustreError: 58411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1937.370142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.826314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.693897] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 1950.222435] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1956.029837] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1958.371934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1958.390638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.409871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1960.425202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.431870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.540187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1965.558718] Lustre: Skipped 2 previous similar messages [ 1970.660594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1970.665977] Lustre: Skipped 1 previous similar message [ 1972.192308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1972.367456] Lustre: server umount lustre-OST0000 complete [ 1994.210401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1994.215666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.225801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.263778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.211516] Lustre: server umount lustre-MDT0000 complete [ 2000.424641] LustreError: 59427:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.444768] LustreError: 59427:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2003.782517] LustreError: 58392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866484 with bad export cookie 3025034770987289501 [ 2003.795736] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2018.273247] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2018.647087] Lustre: server umount lustre-MDT0001 complete [ 2029.383930] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2032.441228] Key type lgssc unregistered [ 2032.769027] LNet: 62137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.778331] LNetError: 62137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.796697] LNet: Removed LNI 192.168.203.150@tcp [ 2033.861765] Key type .llcrypt unregistered [ 2033.867316] Key type ._llcrypt unregistered [ 2058.450558] Key type ._llcrypt registered [ 2058.453693] Key type .llcrypt registered [ 2058.573033] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2059.478507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2059.684929] alg: No test for adler32 (adler32-zlib) [ 2060.927454] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2061.211466] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2062.928166] Key type lgssc registered [ 2064.193664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.723818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.746105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.236056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.597432] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2083.814885] LustreError: 63550:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.848689] LustreError: 63550:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2087.270530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.691852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2092.731387] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2101.428714] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.006571] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2114.991286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2115.416823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.528183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2121.857494] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2129.904647] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2148.337353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2148.343640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.357838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.363340] Lustre: Skipped 1 previous similar message [ 2148.388093] Lustre: Skipped 1 previous similar message [ 2153.446871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.452745] Lustre: Skipped 1 previous similar message [ 2154.210663] Lustre: server umount lustre-OST0000 complete [ 2158.569121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.574620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2158.585723] Lustre: Skipped 1 previous similar message [ 2158.607289] Lustre: Skipped 1 previous similar message [ 2163.682572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.690317] Lustre: Skipped 1 previous similar message [ 2168.805855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.815472] Lustre: Skipped 1 previous similar message [ 2172.384113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2172.577360] Lustre: server umount lustre-MDT0000 complete [ 2173.921611] LustreError: 63551:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.955082] LustreError: 63551:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2175.817054] LustreError: 63532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866656 with bad export cookie 4147936505698668352 [ 2175.821423] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.831091] LustreError: 63532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2176.059225] Lustre: server umount lustre-MDT0001 complete [ 2186.126841] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2189.339748] Key type lgssc unregistered [ 2189.676062] LNet: 67111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.682963] LNetError: 67111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.704104] LNet: Removed LNI 192.168.203.150@tcp [ 2190.822879] Key type .llcrypt unregistered [ 2190.826622] Key type ._llcrypt unregistered [ 2217.064411] Key type ._llcrypt registered [ 2217.066924] Key type .llcrypt registered [ 2217.152977] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2218.195492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.232954] alg: No test for adler32 (adler32-zlib) [ 2219.502482] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2219.988534] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2221.784913] Key type lgssc registered [ 2223.313357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2235.187725] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:38:34 (1787866714) [ 2242.939249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.994477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.501452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.058000] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2254.824374] LustreError: 68816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.850813] LustreError: 68816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2258.431628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.985687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2262.991579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2270.899886] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2277.335112] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2283.823460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2284.173754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.328908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2289.816230] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2297.818417] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2306.316836] Lustre: Failing over lustre-MDT0000 [ 2306.618346] Lustre: server umount lustre-MDT0000 complete [ 2309.097272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.112743] LustreError: 69853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.114447] Lustre: Skipped 1 previous similar message [ 2309.145753] LustreError: 69853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2314.210405] LustreError: 68812:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.232071] LustreError: 68812:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2315.775801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.864228] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.005693] Lustre: *** cfs_fail_loc=135, val=0*** [ 2316.151886] LustreError: 71905:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2316.159428] LustreError: 71905:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -2 [ 2316.169891] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2316.169891] [ 2316.178115] LustreError: MGC192.168.203.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2316.203714] LustreError: 71893:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2316.213169] LustreError: 71893:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2316.234994] LustreError: 71893:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2316.241289] LustreError: 71893:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2316.266806] Lustre: server umount lustre-MDT0000 complete [ 2316.272702] LustreError: 71893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2319.331609] LustreError: 68810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.369031] LustreError: 68810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2322.868931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.034556] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.045150] LustreError: 68807:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2323.400790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.470586] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2325.058857] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2328.263636] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2328.572555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2328.610104] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2328.665392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2338.798496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.807791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2338.816489] Lustre: Skipped 2 previous similar messages [ 2338.827670] Lustre: Skipped 1 previous similar message [ 2342.209128] Lustre: server umount lustre-OST0000 complete [ 2349.026321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.032469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.034352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.883449] Lustre: server umount lustre-MDT0000 complete [ 2354.150818] LustreError: 72731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.180380] LustreError: 72731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2355.976030] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866836 with bad export cookie 13154176555561911611 [ 2355.977458] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2356.003494] LustreError: 68795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2356.476468] Lustre: server umount lustre-MDT0001 complete [ 2368.666940] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2371.482594] Key type lgssc unregistered [ 2371.813557] LNet: 73691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.823491] LNetError: 73691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.849239] LNet: Removed LNI 192.168.203.150@tcp [ 2372.915487] Key type .llcrypt unregistered [ 2372.918182] Key type ._llcrypt unregistered [ 2399.726068] Key type ._llcrypt registered [ 2399.730878] Key type .llcrypt registered [ 2399.860306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2400.899087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.929611] alg: No test for adler32 (adler32-zlib) [ 2401.983214] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2402.332829] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2404.008847] Key type lgssc registered [ 2405.218625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2412.908842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2412.950555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.537484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.313948] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2424.803635] LustreError: 75104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.820374] LustreError: 75104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2427.700402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.179786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2432.321747] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2440.238422] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2446.072634] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2452.953810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2453.338592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2459.520328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2460.012772] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2468.962508] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2475.661747] Lustre: Failing over lustre-MDT0000 [ 2476.053947] Lustre: server umount lustre-MDT0000 complete [ 2476.690519] LustreError: 76344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2479.599756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2479.610791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.631891] LustreError: 75104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.781812] LustreError: 76344:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.797118] LustreError: 76344:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2485.612269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2485.722560] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.809943] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2485.956434] LustreError: 78193:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2485.963765] LustreError: 78193:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -9 [ 2485.976603] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2485.976603] [ 2485.983252] LustreError: MGC192.168.203.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2485.998195] LustreError: 78181:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2486.006159] LustreError: 78181:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2486.013887] LustreError: 78181:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2486.018621] LustreError: 78181:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2486.041540] Lustre: server umount lustre-MDT0000 complete [ 2486.048288] LustreError: 78181:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2486.893235] LustreError: 75099:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.937405] LustreError: 75099:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2491.619183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.768175] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.947589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.006426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2492.025527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2495.375949] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2497.015858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2497.080903] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2497.179514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2505.698484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2505.713046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.723435] Lustre: Skipped 2 previous similar messages [ 2505.733557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.238935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.241925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2510.480642] Lustre: server umount lustre-OST0000 complete [ 2517.476337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2517.478506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.483969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.483976] Lustre: Skipped 1 previous similar message [ 2517.541713] Lustre: Skipped 1 previous similar message [ 2520.901810] Lustre: server umount lustre-MDT0000 complete [ 2522.593051] LustreError: 75832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.614135] LustreError: 75832:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2524.841969] LustreError: 75086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867005 with bad export cookie 17709712219332226030 [ 2524.851790] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2524.864935] LustreError: 75086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2525.187307] Lustre: server umount lustre-MDT0001 complete [ 2536.340868] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2538.984898] Key type lgssc unregistered [ 2539.284583] LNet: 79975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.299914] LNetError: 79975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.310502] LNet: Removed LNI 192.168.203.150@tcp [ 2539.961703] Key type .llcrypt unregistered [ 2539.962991] Key type ._llcrypt unregistered [ 2552.228380] Key type ._llcrypt registered [ 2552.233346] Key type .llcrypt registered [ 2552.400516] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2552.819408] Key type .llcrypt unregistered [ 2552.821218] Key type ._llcrypt unregistered [ 2580.513643] Key type ._llcrypt registered [ 2580.518246] Key type .llcrypt registered [ 2580.627542] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2581.810506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.898059] alg: No test for adler32 (adler32-zlib) [ 2582.933664] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2583.261638] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2584.952167] Key type lgssc registered [ 2586.075672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2594.622851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2594.671557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.249514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.625936] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2606.565969] LustreError: 81880:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.586543] LustreError: 81880:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2609.648242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.048788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2613.889213] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2621.385366] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2628.034348] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2634.527348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2634.734044] LustreError: 83704:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.928622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.218969] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2641.410372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2649.146580] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2660.834554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.847336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.877583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.860619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.870739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.872877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.284760] Lustre: server umount lustre-OST0000 complete [ 2671.586028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2671.598764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.614467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.626088] Lustre: Skipped 1 previous similar message [ 2674.735914] Lustre: server umount lustre-MDT0000 complete [ 2677.218641] LustreError: 84376:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.242812] LustreError: 84376:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2678.619290] LustreError: 83727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867159 with bad export cookie 12861437531261293975 [ 2678.623777] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2678.634373] LustreError: 83727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2678.930231] Lustre: server umount lustre-MDT0001 complete [ 2689.490679] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2691.947185] Key type lgssc unregistered [ 2692.225963] LNet: 85423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2692.235297] LNetError: 85423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2692.250171] LNet: Removed LNI 192.168.203.150@tcp [ 2693.088232] Key type .llcrypt unregistered [ 2693.090987] Key type ._llcrypt unregistered [ 2716.153032] Key type ._llcrypt registered [ 2716.155655] Key type .llcrypt registered [ 2716.263747] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2717.066892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2717.137848] alg: No test for adler32 (adler32-zlib) [ 2718.145366] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2718.396292] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2720.080184] Key type lgssc registered [ 2721.336601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2729.360857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2729.391606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.914517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2735.290489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2740.205270] LustreError: 86838:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.228204] LustreError: 86838:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2743.170631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.506646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.129352] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2756.388880] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2761.788159] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2769.140845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.335769] LustreError: 88659:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.684502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.077320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2776.419343] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2784.462713] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2791.393708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2791.400463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.402796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2791.412883] Lustre: Skipped 1 previous similar message [ 2792.953945] Lustre: server umount lustre-OST0000 complete [ 2800.098517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.110430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.136933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.148276] Lustre: Skipped 1 previous similar message [ 2801.633259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2801.637572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2806.754430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2806.761239] Lustre: Skipped 2 previous similar messages [ 2811.360250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2811.592664] Lustre: server umount lustre-MDT0000 complete [ 2811.882925] LustreError: 86833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.899738] LustreError: 86833:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2815.159415] LustreError: 86819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867296 with bad export cookie 18333581119416730681 [ 2815.173877] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.175394] LustreError: 86819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2815.492379] Lustre: server umount lustre-MDT0001 complete [ 2826.061992] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2828.790262] Key type lgssc unregistered [ 2829.054186] LNet: 90374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.066361] LNetError: 90374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.089164] LNet: Removed LNI 192.168.203.150@tcp [ 2829.993084] Key type .llcrypt unregistered [ 2829.995307] Key type ._llcrypt unregistered [ 2854.652184] Key type ._llcrypt registered [ 2854.654020] Key type .llcrypt registered [ 2854.740561] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 2855.601767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2855.690226] alg: No test for adler32 (adler32-zlib) [ 2856.746327] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 2856.921172] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 2858.616381] Key type lgssc registered [ 2859.745533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2866.787438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2866.801718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.277926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.579449] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2878.434599] LustreError: 91784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.451201] LustreError: 91784:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2880.419031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.769002] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2884.824896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2893.200295] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2899.600640] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2907.553389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.001538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.153551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2914.062983] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 2921.725726] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2936.803036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2936.812327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.822621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.897886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.905208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.919135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.667143] Lustre: server umount lustre-OST0000 complete [ 2947.556827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2947.572248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.605358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.609116] Lustre: Skipped 1 previous similar message [ 2951.146078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.151131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2951.176116] Lustre: Skipped 1 previous similar message [ 2956.257223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.268800] Lustre: Skipped 1 previous similar message [ 2961.376636] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2961.699563] Lustre: server umount lustre-MDT0000 complete [ 2965.962524] LustreError: 91765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867446 with bad export cookie 696241523613193328 [ 2965.968582] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.971809] LustreError: 91765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2966.548964] Lustre: server umount lustre-MDT0001 complete [ 2980.643908] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 2983.769583] Key type lgssc unregistered [ 2984.101930] LNet: 95343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.120479] LNetError: 95343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.153908] LNet: Removed LNI 192.168.203.150@tcp [ 2985.366759] Key type .llcrypt unregistered [ 2985.376629] Key type ._llcrypt unregistered [ 3009.049569] Key type ._llcrypt registered [ 3009.056575] Key type .llcrypt registered [ 3009.205287] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3010.210896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.249900] alg: No test for adler32 (adler32-zlib) [ 3011.478690] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3011.774716] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3013.489106] Key type lgssc registered [ 3014.668872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3023.911097] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3023.928308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.584682] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3044.704269] LustreError: 96734:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 3071.392154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.893571] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3083.791180] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.138467] Lustre: server umount lustre-OST0000 complete [ 3097.296599] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:52:57 (1787867577) [ 3103.368194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.836741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3107.809461] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3108.844437] LustreError: 98373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.890576] LustreError: 98373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3113.954583] LustreError: 98374:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.791111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.052223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3121.440985] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3133.034589] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3143.260751] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3152.094706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3152.590208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.489377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3166.903448] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3173.045290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3188.706856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.716481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.722783] Lustre: Skipped 1 previous similar message [ 3188.733604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3193.843052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3193.862655] Lustre: Skipped 2 previous similar messages [ 3194.334658] Lustre: server umount lustre-OST0000 complete [ 3198.946507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3198.949953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3198.994137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.008303] Lustre: Skipped 1 previous similar message [ 3204.065915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.071075] Lustre: Skipped 1 previous similar message [ 3205.079220] Lustre: server umount lustre-MDT0000 complete [ 3209.187210] LustreError: 98368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.214316] LustreError: 98368:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3210.176281] LustreError: 98355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867691 with bad export cookie 5074630588684424027 [ 3210.181934] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.192511] LustreError: 98355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3210.921473] Lustre: server umount lustre-MDT0001 complete [ 3222.408444] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3226.152199] Key type lgssc unregistered [ 3226.575350] LNet: 102094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3226.604177] LNetError: 102094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3226.632821] LNet: Removed LNI 192.168.203.150@tcp [ 3227.538069] Key type .llcrypt unregistered [ 3227.541982] Key type ._llcrypt unregistered [ 3269.524201] Key type ._llcrypt registered [ 3269.532531] Key type .llcrypt registered [ 3269.628578] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3270.810138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.914917] alg: No test for adler32 (adler32-zlib) [ 3271.986178] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3272.236550] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3273.888729] Key type lgssc registered [ 3274.937258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3282.836336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3282.878358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3284.379750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.020792] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3294.699931] LustreError: 104280:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3294.724654] LustreError: 104280:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3297.790767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3298.172321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3303.081677] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3310.909654] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3316.913762] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3324.240958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3324.551866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3329.707475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3330.399041] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3337.539676] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3350.502532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3350.518270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.544375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.571996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3353.573153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3353.601671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.769099] Lustre: server umount lustre-OST0000 complete [ 3363.819800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.822393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3363.836562] Lustre: Skipped 1 previous similar message [ 3363.848938] Lustre: Skipped 2 previous similar messages [ 3367.254946] Lustre: server umount lustre-MDT0000 complete [ 3368.929393] LustreError: 104280:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.957586] LustreError: 104280:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3370.779050] LustreError: 104261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867851 with bad export cookie 3235466200664561142 [ 3370.779897] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3370.791058] LustreError: 104261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3371.128952] Lustre: server umount lustre-MDT0001 complete [ 3381.378296] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3383.953114] Key type lgssc unregistered [ 3384.284592] LNet: 107839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.314970] LNetError: 107839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.347966] LNet: Removed LNI 192.168.203.150@tcp [ 3385.388888] Key type .llcrypt unregistered [ 3385.391687] Key type ._llcrypt unregistered [ 3403.800843] Key type ._llcrypt registered [ 3403.803751] Key type .llcrypt registered [ 3403.900558] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3405.302357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3405.347416] alg: No test for adler32 (adler32-zlib) [ 3406.417709] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3406.656945] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3408.368879] Key type lgssc registered [ 3409.805924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3417.431676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3417.453227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3432.995957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3438.112319] LustreError: 109083:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 3464.800218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.338510] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3480.401932] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3487.393358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.705493] LustreError: MGC192.168.203.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3487.716646] LustreError: 110107:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3487.725307] LustreError: 110107:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3487.740049] LustreError: 110107:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3487.857599] Lustre: server umount lustre-MDT0000 complete [ 3487.863930] LustreError: 110107:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3505.322159] Lustre: server umount lustre-OST0000 complete [ 3516.127229] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3527.264507] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3564.912905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.149576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3565.195737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3565.301654] Lustre: lustre-MDT0000: new disk, initializing [ 3565.446834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.497104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3570.906700] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3582.485790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.638509] Lustre: 112897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3582.714790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3582.720128] Lustre: Skipped 1 previous similar message [ 3582.803314] Lustre: lustre-MDT0001: new disk, initializing [ 3582.966468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3583.052825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3583.064430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3587.691302] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3597.210394] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3603.849210] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.066022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.299227] Lustre: lustre-OST0000: new disk, initializing [ 3611.306422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3611.311815] Lustre: 114881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3611.440264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.523187] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3619.489992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3619.509792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3619.650771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3629.069345] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.920807] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3637.275895] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3642.880096] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3643.131333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3650.531302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.542171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.546826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.548108] LustreError: Skipped 1 previous similar message [ 3650.559455] Lustre: Skipped 1 previous similar message [ 3651.933508] Lustre: server umount lustre-OST0000 complete [ 3655.650063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.662701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.668615] Lustre: Skipped 2 previous similar messages [ 3660.771143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.792462] Lustre: Skipped 1 previous similar message [ 3661.952889] Lustre: server umount lustre-MDT0000 complete [ 3665.422190] LustreError: 114880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868146 with bad export cookie 764546478777493156 [ 3665.437098] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3665.438291] LustreError: 114880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3665.783250] Lustre: server umount lustre-MDT0001 complete [ 3674.218909] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:02:33 (1787868153) [ 3677.532891] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3694.257883] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3703.582377] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3742.095059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.366555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3742.402563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3742.502531] Lustre: lustre-MDT0000: new disk, initializing [ 3742.596004] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3742.631364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3747.332618] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3758.211730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.377162] Lustre: 119784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3758.459391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3758.471460] Lustre: Skipped 1 previous similar message [ 3758.595872] Lustre: lustre-MDT0001: new disk, initializing [ 3758.730189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3758.810111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3758.827176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3763.751118] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3773.324642] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3779.459709] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3786.976867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3787.336780] Lustre: lustre-OST0000: new disk, initializing [ 3787.345796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3787.363426] Lustre: 121754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3787.505530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.134538] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3796.670910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3796.702061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3796.880759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3806.868483] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3812.439931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3812.663713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3816.732416] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3816.952515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3822.566919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.567833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.597972] Lustre: Skipped 1 previous similar message [ 3822.610437] LustreError: Skipped 1 previous similar message [ 3822.612305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.633373] Lustre: Skipped 1 previous similar message [ 3824.710056] Lustre: server umount lustre-OST0000 complete [ 3830.243405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3830.253394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.283570] Lustre: Skipped 1 previous similar message [ 3830.299541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.576693] Lustre: server umount lustre-MDT0000 complete [ 3839.218316] LustreError: 121752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868320 with bad export cookie 764546478777494038 [ 3839.229844] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.598976] Lustre: server umount lustre-MDT0001 complete [ 3847.865329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.324842] LustreError: 123969:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.351461] LustreError: 123969:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3848.406024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.748142] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3853.795212] LustreError: 123970:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.919320] LustreError: 123969:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.372147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.186340] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3873.472936] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3879.702822] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3886.127934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.461579] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 3899.311419] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3906.230298] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3913.189861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3913.197633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.207979] Lustre: Skipped 1 previous similar message [ 3913.212772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.217682] Lustre: Skipped 2 previous similar messages [ 3916.385981] Lustre: server umount lustre-OST0000 complete [ 3923.437064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3923.456041] LustreError: Skipped 1 previous similar message [ 3923.462706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.489586] Lustre: Skipped 1 previous similar message [ 3929.571478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3929.574408] Lustre: Skipped 5 previous similar messages [ 3935.201384] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3935.356716] Lustre: server umount lustre-MDT0000 complete [ 3938.787324] LustreError: 125814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868419 with bad export cookie 764546478777494920 [ 3938.790968] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.793130] LustreError: 125814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3939.054533] Lustre: server umount lustre-MDT0001 complete [ 3948.873993] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 3951.195087] Key type lgssc unregistered [ 3951.389731] LNet: 127602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3951.393607] LNetError: 127602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3951.422404] LNet: Removed LNI 192.168.203.150@tcp [ 3952.552261] Key type .llcrypt unregistered [ 3952.555411] Key type ._llcrypt unregistered [ 3970.196174] Key type ._llcrypt registered [ 3970.202338] Key type .llcrypt registered [ 3970.349181] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 3984.956877] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 3985.882037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3986.107555] alg: No test for adler32 (adler32-zlib) [ 3987.160054] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 3987.424600] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 3989.105494] Key type lgssc registered [ 3990.458836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4024.826194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4024.847104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.139132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4026.162939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4026.265239] Lustre: lustre-MDT0000: new disk, initializing [ 4026.373093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4026.397837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4031.378932] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4041.917972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.146866] Lustre: 130757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4042.223905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4042.237132] Lustre: Skipped 1 previous similar message [ 4042.396141] Lustre: lustre-MDT0001: new disk, initializing [ 4042.531806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4042.565018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4042.586885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4047.511539] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4059.282733] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4065.012382] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4072.592041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4072.863102] Lustre: lustre-OST0000: new disk, initializing [ 4072.865464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4072.871945] Lustre: 132725:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4072.960364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.000638] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4082.243466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4082.256710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4082.364721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4087.109845] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4091.248931] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4091.502434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4096.028952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4096.299265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4098.029686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.035263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.049134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4103.138680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.140146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4108.268976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4108.282213] Lustre: Skipped 2 previous similar messages [ 4112.352267] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4112.520932] Lustre: server umount lustre-OST0000 complete [ 4118.500589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.523177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.616485] Lustre: server umount lustre-MDT0000 complete [ 4123.627691] LustreError: 130769:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.653524] LustreError: 130769:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4126.146726] LustreError: 130750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868607 with bad export cookie 1145054953473058104 [ 4126.150169] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4126.167917] LustreError: 130750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4126.552521] Lustre: server umount lustre-MDT0001 complete [ 4134.547111] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:10:14 (1787868614) [ 4140.808257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.380763] LustreError: 135140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.529783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.204551] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4146.657954] LustreError: 135141:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.780231] LustreError: 135140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.056563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.439677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4158.866395] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4166.360840] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4172.140326] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4174.821613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4174.828440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.842387] Lustre: Skipped 1 previous similar message [ 4174.849029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.863584] Lustre: Skipped 1 previous similar message [ 4180.705453] Lustre: server umount lustre-MDT0000 complete [ 4184.653896] LustreError: 135120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868665 with bad export cookie 1145054953473058993 [ 4184.658151] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4184.665071] LustreError: 135120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4185.026808] Lustre: server umount lustre-MDT0001 complete [ 4192.947914] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:11:12 (1787868672) [ 4200.087880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4215.714264] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4220.837439] LustreError: 137557:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 4246.625588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4252.121599] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4259.616644] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4261.720457] Lustre: server umount lustre-OST0000 complete [ 4269.363666] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:12:29 (1787868749) [ 4274.919332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.382314] LustreError: 139003:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.490440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.936874] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4285.923381] LustreError: 139003:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.961163] LustreError: 139003:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4288.457859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.606278] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4300.580372] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4305.749942] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4311.912137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.293990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.302544] Lustre: Skipped 1 previous similar message [ 4318.593628] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4326.345822] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4335.077101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4335.084674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4335.098625] Lustre: Skipped 1 previous similar message [ 4335.104174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4335.109167] Lustre: Skipped 4 previous similar messages [ 4339.931676] Lustre: server umount lustre-MDT0000 complete [ 4341.230938] LustreError: 138998:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.272780] LustreError: 138998:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4343.473073] LustreError: 138984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868824 with bad export cookie 1145054953473059658 [ 4343.478557] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4343.484937] LustreError: 138984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4343.894920] Lustre: server umount lustre-MDT0001 complete [ 4357.904326] Lustre: server umount lustre-OST0000 complete [ 4366.099859] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:14:06 (1787868846) [ 4372.550522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.358659] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4378.084829] LustreError: 142665:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.101033] LustreError: 142665:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4387.032915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.403491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.412105] Lustre: Skipped 1 previous similar message [ 4391.727533] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4399.550532] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4405.967619] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4413.628302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4418.032630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4419.613561] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4426.331451] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.939741] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4430.112926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4434.157616] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4434.481093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4437.477482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.478438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4437.480950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 4437.494899] Lustre: Skipped 3 previous similar messages [ 4437.519028] Lustre: Skipped 4 previous similar messages [ 4442.180920] Lustre: server umount lustre-OST0000 complete [ 4451.960556] Lustre: server umount lustre-MDT0000 complete [ 4452.837150] LustreError: 142664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.857774] LustreError: 142664:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4455.464840] LustreError: 142647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868936 with bad export cookie 1145054953473060722 [ 4455.472774] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.479068] LustreError: 142647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4455.505595] LustreError: 143394:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 4455.519620] LustreError: 143394:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 4455.819971] Lustre: server umount lustre-MDT0001 complete [ 4463.737800] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:15:43 (1787868943) [ 4470.773976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4486.368634] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4491.488453] LustreError: 146820:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 4517.280499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.288633] Lustre: Skipped 1 previous similar message [ 4524.654139] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4533.023825] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4541.279965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.112698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4546.651069] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4555.371902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.063952] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4567.677423] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4573.120895] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4577.400546] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4577.628301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4581.205691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4581.367286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.478896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.488972] Lustre: Skipped 2 previous similar messages [ 4586.489794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.490846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.490850] Lustre: Skipped 2 previous similar messages [ 4586.530137] LustreError: Skipped 1 previous similar message [ 4589.286396] Lustre: server umount lustre-MDT0000 complete [ 4591.588697] LustreError: 147857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.601889] LustreError: 147857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4592.597238] LustreError: 146826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869073 with bad export cookie 1145054953473061751 [ 4592.612463] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4593.023496] Lustre: server umount lustre-MDT0001 complete [ 4606.796700] Lustre: server umount lustre-OST0000 complete [ 4614.324817] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:18:13 (1787869093) [ 4621.342416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4637.024513] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4642.144529] LustreError: 151015:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 4673.276673] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4681.327950] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4689.179769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.909684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4694.660112] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4703.482180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.898443] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4716.442853] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4723.615382] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4731.065763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.245513] Lustre: lustre-OST0001: new disk, initializing [ 4731.249898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4731.254886] Lustre: 153887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4738.091369] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4741.228495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4741.243949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4741.370745] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4748.936831] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4754.104667] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4754.317969] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4759.598631] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4759.790464] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4762.087385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4762.100501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4762.115537] Lustre: Skipped 2 previous similar messages [ 4762.129880] Lustre: Skipped 2 previous similar messages [ 4768.123182] Lustre: server umount lustre-OST0000 complete [ 4772.322386] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.331930] LustreError: Skipped 1 previous similar message [ 4778.298355] Lustre: server umount lustre-OST0001 complete [ 4788.966712] Lustre: server umount lustre-MDT0000 complete [ 4792.711309] LustreError: 151022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869273 with bad export cookie 1145054953473062703 [ 4792.719267] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4792.723979] LustreError: 151022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4793.267291] Lustre: server umount lustre-MDT0001 complete [ 4820.812208] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:21:40 (1787869300) [ 4822.933931] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4824.900529] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4826.756990] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:21:46 (1787869306) [ 4834.384377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.778596] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4834.819526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4834.961249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4834.967371] Lustre: Skipped 6 previous similar messages [ 4838.981549] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4847.460674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.535832] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4847.554570] Lustre: 157448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4852.104667] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4859.965563] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4866.292081] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4872.657572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.916514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4878.780947] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4879.506288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4885.279932] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4889.622428] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4889.849827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4894.297896] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4894.458486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4899.812685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4899.821665] LustreError: Skipped 1 previous similar message [ 4899.828796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.843120] Lustre: Skipped 4 previous similar messages [ 4899.849862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4899.856253] Lustre: Skipped 9 previous similar messages [ 4902.789765] Lustre: server umount lustre-OST0000 complete [ 4925.110947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4926.511284] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.520750] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4933.291304] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 4940.838405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.877406] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4942.235510] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4948.358119] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4948.568964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4953.447894] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4953.689424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4960.289652] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c758446f800.ost_server_uuid 50 [ 4961.762774] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c758446f800.ost_server_uuid in FULL state after 0 sec [ 4961.957103] LustreError: 159317:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.50@tcp) returned error from blocking AST (req@0000000005b00e76 x1874715966959360 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000005e6beaa2/0xfe40d69d6104089 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.50@tcp remote: 0x59f097c4ebdb0f09 expref: 5 pid: 157456 timeout: 5061 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4962.018551] LustreError: lustre-MDT0000: A client on nid 192.168.203.50@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4962.029752] LustreError: 157445:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.203.50@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000005e6beaa2/0xfe40d69d6104089 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.50@tcp remote: 0x59f097c4ebdb0f09 expref: 6 pid: 157456 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4966.376918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4966.390333] LustreError: Skipped 1 previous similar message [ 4972.080631] Lustre: server umount lustre-OST0000 complete [ 4983.830439] Lustre: server umount lustre-MDT0000 complete [ 4986.851511] LustreError: 158234:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.875306] LustreError: 158234:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 4987.372182] LustreError: 157442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869468 with bad export cookie 1145054953473063886 [ 4987.373384] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4987.381860] LustreError: 157442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4993.747705] Lustre: server umount lustre-MDT0001 complete [ 5005.464271] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5007.866264] Key type lgssc unregistered [ 5008.179542] LNet: 163568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5008.204848] LNetError: 163568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5008.229706] LNet: Removed LNI 192.168.203.150@tcp [ 5009.068196] Key type .llcrypt unregistered [ 5009.072511] Key type ._llcrypt unregistered [ 5031.261570] Key type ._llcrypt registered [ 5031.263746] Key type .llcrypt registered [ 5031.343772] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5032.204223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5032.328553] alg: No test for adler32 (adler32-zlib) [ 5033.426549] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5033.639945] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5035.320149] Key type lgssc registered [ 5036.445551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5043.814881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5043.837594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.383143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.095922] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5055.973065] LustreError: 164982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5055.988914] LustreError: 164982:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5057.689646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.036062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5062.218237] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5071.057488] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5077.277084] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5083.808329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.235098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.948945] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5090.436208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5097.769277] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5107.019703] Lustre: Failing over lustre-MDT0000 [ 5107.341314] Lustre: server umount lustre-MDT0000 complete [ 5108.199747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5108.213735] LustreError: 164982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.222239] Lustre: Skipped 1 previous similar message [ 5113.316147] LustreError: 166019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5113.342724] LustreError: 166019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5118.441206] LustreError: 164983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.453431] LustreError: 164983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5119.677352] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.50@tcp arrived at 1787869600 with bad export cookie 14902321563322283131 [ 5119.697358] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5123.557670] LustreError: 164982:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.575919] LustreError: 164982:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5124.576522] Lustre: 164294:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869589/real 1787869589] req@000000006897c1a8 x1874717063540352/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787869605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5124.602555] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5126.983497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5134.823066] LustreError: 164983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.850196] LustreError: 164983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5135.112212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.175415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5139.768665] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5140.460720] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5140.473577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5142.080204] Lustre: lustre-MDT0000: Denying connection for new client b4461ad6-8b3a-4b01-ae54-a534763e0f50 (at 192.168.203.50@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5150.695066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.702884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.724258] Lustre: Skipped 2 previous similar messages [ 5150.739706] Lustre: Skipped 1 previous similar message [ 5155.813493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5155.821900] Lustre: Skipped 1 previous similar message [ 5156.089311] Lustre: server umount lustre-OST0000 complete [ 5159.903191] LustreError: 168714:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5159.912522] Lustre: 168063:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5159.923533] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5159.944530] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5159.952520] LustreError: 168063:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009519b4ff x1874717063560320/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5159.967539] LustreError: 168063:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5159.975509] LustreError: 168063:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5159.988103] Lustre: lustre-MDT0000: Recovery over after 0:19, of 2 clients 0 recovered and 2 were evicted. [ 5160.341508] Lustre: server umount lustre-MDT0000 complete [ 5160.929773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.946975] LustreError: 164978:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5163.759035] LustreError: 166824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869644 with bad export cookie 14902321563322283236 [ 5163.765943] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5163.769480] LustreError: 166824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5170.236168] Lustre: server umount lustre-MDT0001 complete [ 5182.378483] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5185.480552] Key type lgssc unregistered [ 5185.799651] LNet: 169470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5185.808303] LNetError: 169470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5185.826728] LNet: Removed LNI 192.168.203.150@tcp [ 5186.825362] Key type .llcrypt unregistered [ 5186.831571] Key type ._llcrypt unregistered [ 5210.959260] Key type ._llcrypt registered [ 5210.965219] Key type .llcrypt registered [ 5211.090489] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5212.740865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5212.779787] alg: No test for adler32 (adler32-zlib) [ 5213.894148] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5214.102215] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5215.728231] Key type lgssc registered [ 5216.717855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5223.521650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5223.576516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.079245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.057148] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5235.692210] LustreError: 170883:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.712261] LustreError: 170883:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5237.546824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.927449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.706058] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5250.267945] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5256.080423] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5263.135074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.451822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5269.843935] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5270.658627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5277.855978] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5286.370607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5286.378952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.395332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5288.426028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5288.430292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5288.434489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5291.610974] Lustre: server umount lustre-OST0000 complete [ 5298.657927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.675892] Lustre: Skipped 1 previous similar message [ 5298.676386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5298.696788] Lustre: Skipped 2 previous similar messages [ 5303.778142] LustreError: 170878:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5303.807242] LustreError: 170878:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5304.103654] Lustre: server umount lustre-MDT0000 complete [ 5307.791310] LustreError: 171620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869788 with bad export cookie 10209512740354160995 [ 5307.801303] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5307.804524] LustreError: 171620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5308.152025] Lustre: server umount lustre-MDT0001 complete [ 5319.937136] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5322.694810] Key type lgssc unregistered [ 5322.973495] LNet: 174418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5322.986323] LNetError: 174418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5324.013179] LNet: Removed LNI 192.168.203.150@tcp [ 5324.996996] Key type .llcrypt unregistered [ 5324.998900] Key type ._llcrypt unregistered [ 5358.431599] Key type ._llcrypt registered [ 5358.434299] Key type .llcrypt registered [ 5358.529559] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5359.785627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5359.838177] alg: No test for adler32 (adler32-zlib) [ 5361.035611] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5361.379789] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5363.074933] Key type lgssc registered [ 5364.261623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5372.023897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5372.040940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5373.486110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5377.274906] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5383.650111] LustreError: 176119:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.666202] LustreError: 176119:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5386.680545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.462471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5395.051191] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5406.027636] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5412.181131] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.945166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.247245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.276942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5425.687871] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5434.291120] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5443.048586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.053025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5443.053391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.078479] Lustre: Skipped 1 previous similar message [ 5447.267590] Lustre: server umount lustre-OST0000 complete [ 5453.282828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.289449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5453.298777] Lustre: Skipped 1 previous similar message [ 5453.314921] Lustre: Skipped 2 previous similar messages [ 5458.408479] LustreError: 176114:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.440785] LustreError: 176114:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5458.681237] Lustre: server umount lustre-MDT0000 complete [ 5463.252668] LustreError: 176102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869944 with bad export cookie 5811712467946544894 [ 5463.258713] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5463.265277] LustreError: 176102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5463.519348] Lustre: server umount lustre-MDT0001 complete [ 5477.163335] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5479.819306] Key type lgssc unregistered [ 5480.121603] LNet: 179659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5480.128676] LNetError: 179659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5480.149127] LNet: Removed LNI 192.168.203.150@tcp [ 5481.070599] Key type .llcrypt unregistered [ 5481.075334] Key type ._llcrypt unregistered [ 5501.064900] Key type ._llcrypt registered [ 5501.067496] Key type .llcrypt registered [ 5501.204380] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5502.436429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5502.498584] alg: No test for adler32 (adler32-zlib) [ 5503.749805] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5504.030220] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5505.704407] Key type lgssc registered [ 5507.312203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5518.211452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5518.235543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.617279] Lustre: *** cfs_fail_loc=135, val=0*** [ 5519.698457] LustreError: 181163:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5519.709364] LustreError: 181163:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.150@tcp: cfg command failed: rc = -2 [ 5519.722952] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5519.722952] [ 5519.730773] LustreError: MGC192.168.203.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5519.743494] LustreError: 181125:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5519.748584] LustreError: 181125:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5519.765220] LustreError: 181125:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5519.775564] LustreError: 181125:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5519.835115] Lustre: server umount lustre-MDT0000 complete [ 5519.840199] LustreError: 181125:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5526.904779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.359159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.868636] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5537.764216] LustreError: 181533:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.799813] LustreError: 181533:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5540.669083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.964257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5545.428836] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5552.848819] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5558.991567] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5561.327641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5561.340575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.362687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.850760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.864965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.873618] Lustre: Skipped 1 previous similar message [ 5565.926206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5565.934209] Lustre: Skipped 1 previous similar message [ 5567.407216] Lustre: server umount lustre-MDT0000 complete [ 5571.043117] LustreError: 181529:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.068540] LustreError: 181529:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5571.511489] LustreError: 181515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870052 with bad export cookie 16119997118108481075 [ 5571.520730] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5571.525447] LustreError: 181515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5571.867976] Lustre: server umount lustre-MDT0001 complete [ 5583.085310] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5585.720830] Key type lgssc unregistered [ 5585.954090] LNet: 183861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5585.960728] LNetError: 183861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5586.985617] LNet: Removed LNI 192.168.203.150@tcp [ 5587.931050] Key type .llcrypt unregistered [ 5587.934224] Key type ._llcrypt unregistered [ 5610.280478] Key type ._llcrypt registered [ 5610.282795] Key type .llcrypt registered [ 5610.375725] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5610.689392] Key type .llcrypt unregistered [ 5610.692832] Key type ._llcrypt unregistered [ 5630.832717] Key type ._llcrypt registered [ 5630.835588] Key type .llcrypt registered [ 5630.934050] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5632.146526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5632.187170] alg: No test for adler32 (adler32-zlib) [ 5633.376285] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5633.665199] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5635.361097] Key type lgssc registered [ 5636.632878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5645.232734] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5645.277116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5660.896387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5666.016272] LustreError: 185998:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.150@tcp: failed processing log, type 4: rc = -110 [ 5692.772098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.548649] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5707.819025] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5715.392964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.898807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5715.985346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5720.747791] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5721.061554] LustreError: 186023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.074242] LustreError: 186023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5726.190852] LustreError: 186024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.203788] LustreError: 186023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.230517] LustreError: 186023:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5728.944032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.352668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5733.674239] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5741.547534] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5747.371300] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5750.339927] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5762.017976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.037087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.051229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.093087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.102342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.105401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.131266] Lustre: Skipped 1 previous similar message [ 5770.210983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.330306] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5775.335404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.359252] Lustre: Skipped 2 previous similar messages [ 5775.631088] Lustre: server umount lustre-OST0000 complete [ 5780.449265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5780.454442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.465700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.498252] Lustre: Skipped 1 previous similar message [ 5785.816828] Lustre: server umount lustre-MDT0000 complete [ 5789.650252] LustreError: 186004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870270 with bad export cookie 12585463102717900338 [ 5789.654274] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5789.664626] LustreError: 186004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5790.079407] Lustre: server umount lustre-MDT0001 complete [ 5801.684671] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 5804.588409] Key type lgssc unregistered [ 5804.947982] LNet: 189971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5804.959780] LNetError: 189971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5804.980489] LNet: Removed LNI 192.168.203.150@tcp [ 5805.774542] Key type .llcrypt unregistered [ 5805.776877] Key type ._llcrypt unregistered [ 5830.051492] Key type ._llcrypt registered [ 5830.062586] Key type .llcrypt registered [ 5830.226822] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 5831.528414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.560898] alg: No test for adler32 (adler32-zlib) [ 5832.684426] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 5832.920412] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 5834.640167] Key type lgssc registered [ 5835.613872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.883153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.922562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.464580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.062653] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5855.746356] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.787947] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5860.851704] LustreError: 191389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.884704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.436622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5868.959060] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5876.812710] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5883.123844] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5890.522628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5890.916312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.541337] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5906.564902] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5915.453328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5921.942236] Lustre: Failing over lustre-MDT0000 [ 5923.476646] LustreError: 192448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.271217] Lustre: server umount lustre-MDT0000 complete [ 5925.857926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5925.867696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.878189] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.895270] Lustre: Skipped 2 previous similar messages [ 5925.936099] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5930.982351] LustreError: 191389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.022884] LustreError: 191389:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5939.843387] LustreError: 191385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.859292] LustreError: 191385:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5941.219917] Lustre: 190698:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787870406/real 1787870406] req@0000000067f65765 x1874717901354368/t0(0) o400->MGC192.168.203.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787870422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5941.272795] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.766087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.461576] LustreError: 190694:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000017bc5eca x1874717901359104/t0(0) o250->MGC192.168.203.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5951.819958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.882695] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5954.150518] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5957.115502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5957.142138] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5957.198923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5957.320893] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 5961.919478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5966.217330] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5975.014210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5975.030735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.057431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.039388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.052667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.070703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5976.794337] Lustre: server umount lustre-OST0000 complete [ 5982.689416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5982.691854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.700645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.719693] Lustre: Skipped 1 previous similar message [ 5987.401059] Lustre: server umount lustre-MDT0000 complete [ 5987.810816] LustreError: 191389:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.833811] LustreError: 191389:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 5991.321904] LustreError: 191368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870472 with bad export cookie 7987686615018219206 [ 5991.327505] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5991.339042] LustreError: 191368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5991.813548] Lustre: server umount lustre-MDT0001 complete [ 6003.579426] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6007.303989] Key type lgssc unregistered [ 6007.610661] LNet: 196206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6007.625836] LNetError: 196206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6007.669170] LNet: Removed LNI 192.168.203.150@tcp [ 6009.033600] Key type .llcrypt unregistered [ 6009.036191] Key type ._llcrypt unregistered [ 6031.176711] Key type ._llcrypt registered [ 6031.178878] Key type .llcrypt registered [ 6031.296784] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6032.186458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6032.246367] alg: No test for adler32 (adler32-zlib) [ 6033.601679] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6033.962187] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6035.672262] Key type lgssc registered [ 6037.385156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.820659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6047.843322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.480391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6054.899488] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6060.005449] LustreError: 197619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.024335] LustreError: 197619:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6064.023442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.382147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6069.079160] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6077.591721] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6083.564871] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6091.071505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6091.449906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.547627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6097.743450] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6105.104798] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6108.990585] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6115.978247] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6133.426803] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6146.023321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.027449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.065416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6149.920710] Lustre: server umount lustre-OST0000 complete [ 6156.258535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.260841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.267835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.285322] Lustre: Skipped 2 previous similar messages [ 6156.300472] Lustre: Skipped 2 previous similar messages [ 6160.615412] Lustre: server umount lustre-MDT0000 complete [ 6161.380833] LustreError: 199191:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.401383] LustreError: 199191:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6163.600938] LustreError: 197601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870644 with bad export cookie 9223227888161690058 [ 6163.617662] LustreError: 197601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.621401] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.022717] Lustre: server umount lustre-MDT0001 complete [ 6174.708504] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6177.831407] Key type lgssc unregistered [ 6178.143800] LNet: 201492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6178.152586] LNetError: 201492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6179.179836] LNet: Removed LNI 192.168.203.150@tcp [ 6180.030684] Key type .llcrypt unregistered [ 6180.032677] Key type ._llcrypt unregistered [ 6198.788741] Key type ._llcrypt registered [ 6198.795365] Key type .llcrypt registered [ 6198.972167] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6200.312469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6200.388780] alg: No test for adler32 (adler32-zlib) [ 6201.446361] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6201.710575] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6203.384165] Key type lgssc registered [ 6204.598951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6212.870379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6212.895554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.476169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.716824] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6224.866414] LustreError: 202760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.891938] LustreError: 202760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6227.493434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.945054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6232.740225] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6240.752244] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6247.069789] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6254.114164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6254.534401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6259.633154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6260.855591] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6267.903389] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6274.190565] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6288.066364] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6299.297516] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6307.534164] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6319.084483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.090826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.096701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.104022] Lustre: Skipped 1 previous similar message [ 6319.115320] Lustre: Skipped 1 previous similar message [ 6323.821792] Lustre: server umount lustre-OST0000 complete [ 6329.314488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.344194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.354069] Lustre: Skipped 1 previous similar message [ 6334.016178] Lustre: server umount lustre-MDT0000 complete [ 6334.434372] LustreError: 202760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.452672] LustreError: 202760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6338.215994] LustreError: 202740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870819 with bad export cookie 466439060702038243 [ 6338.218378] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.229824] LustreError: 202740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6338.592399] Lustre: server umount lustre-MDT0001 complete [ 6350.391618] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6353.089679] Key type lgssc unregistered [ 6353.361692] LNet: 207783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6353.367984] LNetError: 207783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6353.386666] LNet: Removed LNI 192.168.203.150@tcp [ 6354.261230] Key type .llcrypt unregistered [ 6354.265525] Key type ._llcrypt unregistered [ 6379.742849] Key type ._llcrypt registered [ 6379.747122] Key type .llcrypt registered [ 6379.886500] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6381.205986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6381.263368] alg: No test for adler32 (adler32-zlib) [ 6382.312364] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6382.525896] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6384.168156] Key type lgssc registered [ 6385.384676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6393.510254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6393.537181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.098799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.567691] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6405.602497] LustreError: 209198:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.624165] LustreError: 209198:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6407.390163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6407.843619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6413.776306] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6423.275281] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6430.241594] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.021071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.427510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.497564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6444.069710] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6452.344429] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6468.068435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6468.084187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.094472] Lustre: Skipped 1 previous similar message [ 6468.110120] Lustre: Skipped 1 previous similar message [ 6472.832068] Lustre: server umount lustre-OST0000 complete [ 6478.305552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.323937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6478.331678] Lustre: Skipped 1 previous similar message [ 6478.349958] Lustre: Skipped 1 previous similar message [ 6483.040171] Lustre: server umount lustre-MDT0000 complete [ 6483.427461] LustreError: 209193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.453724] LustreError: 209193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6486.899566] LustreError: 209179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870967 with bad export cookie 18321775364352318712 [ 6486.907299] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6486.917593] LustreError: 209179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6487.298910] Lustre: server umount lustre-MDT0001 complete [ 6499.788955] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6503.101235] Key type lgssc unregistered [ 6503.437861] LNet: 212889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6503.446516] LNetError: 212889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6503.465967] LNet: Removed LNI 192.168.203.150@tcp [ 6504.422305] Key type .llcrypt unregistered [ 6504.426433] Key type ._llcrypt unregistered [ 6523.643945] Key type ._llcrypt registered [ 6523.645938] Key type .llcrypt registered [ 6523.790605] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6525.040699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6525.056638] alg: No test for adler32 (adler32-zlib) [ 6526.166457] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6526.538180] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6528.272170] Key type lgssc registered [ 6529.418871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6537.857099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6537.889352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.305262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.096184] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6549.476340] LustreError: 214156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.500816] LustreError: 214156:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6553.309749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.727645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6558.516445] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6566.382190] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6572.066446] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6579.117317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.580986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.690822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6586.147717] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6594.073357] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.378732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.379785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6609.379981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6609.410772] Lustre: Skipped 1 previous similar message [ 6613.210912] Lustre: server umount lustre-OST0000 complete [ 6619.622747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.637865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6619.644230] Lustre: Skipped 2 previous similar messages [ 6622.775743] Lustre: server umount lustre-MDT0000 complete [ 6624.739572] LustreError: 215398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6624.759329] LustreError: 215398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6626.291618] LustreError: 214136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871107 with bad export cookie 1325809440692841136 [ 6626.295608] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6626.304446] LustreError: 214136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6626.734921] Lustre: server umount lustre-MDT0001 complete [ 6636.956312] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6639.644691] Key type lgssc unregistered [ 6639.917172] LNet: 217843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6639.927869] LNetError: 217843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6639.947606] LNet: Removed LNI 192.168.203.150@tcp [ 6640.808518] Key type .llcrypt unregistered [ 6640.812330] Key type ._llcrypt unregistered [ 6659.511628] Key type ._llcrypt registered [ 6659.513879] Key type .llcrypt registered [ 6659.645517] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6660.744405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.774984] alg: No test for adler32 (adler32-zlib) [ 6661.817332] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6662.076687] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6663.744237] Key type lgssc registered [ 6664.796380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6671.797725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.823363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.418107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.031140] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6683.619702] LustreError: 219115:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.640820] LustreError: 219115:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6686.412363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.690266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6691.181321] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6698.039087] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6703.467359] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6711.292030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6711.635279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.510598] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6718.839184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6725.753514] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6735.333859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6735.378061] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6735.573956] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6742.630833] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6751.593393] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6753.451655] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6765.411192] Lustre: Permanently deactivating lustre-OST0001 [ 6765.432239] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6773.842996] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6779.546521] Lustre: Permanently reactivating lustre-OST0001 [ 6779.583942] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6779.600658] Lustre: Skipped 2 previous similar messages [ 6783.693600] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.706404] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6783.714919] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6783.727382] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6783.782407] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6794.223801] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.226035] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6794.245463] Lustre: Skipped 1 previous similar message [ 6794.250319] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6795.207330] Lustre: server umount lustre-OST0001 complete [ 6799.339413] LustreError: 220939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6799.341513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6799.343135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6799.343140] Lustre: Skipped 1 previous similar message [ 6799.352015] LustreError: 220939:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6799.360551] Lustre: Skipped 1 previous similar message [ 6804.464407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6804.465855] LustreError: 220936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6804.485109] Lustre: Skipped 2 previous similar messages [ 6804.519622] LustreError: 220936:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6804.754753] Lustre: server umount lustre-OST0000 complete [ 6809.569923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6809.572541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.582831] LustreError: Skipped 1 previous similar message [ 6809.584129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.593812] Lustre: Skipped 1 previous similar message [ 6809.623449] Lustre: Skipped 1 previous similar message [ 6814.694507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6814.698147] Lustre: Skipped 1 previous similar message [ 6814.779145] Lustre: server umount lustre-MDT0000 complete [ 6818.689253] LustreError: 219094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871299 with bad export cookie 520086268716384944 [ 6818.691790] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6818.712767] LustreError: 219094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6819.292776] Lustre: server umount lustre-MDT0001 complete [ 6831.533577] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 6834.925704] Key type lgssc unregistered [ 6835.308429] LNet: 224188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6835.318845] LNetError: 224188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6835.369311] LNet: Removed LNI 192.168.203.150@tcp [ 6836.454891] Key type .llcrypt unregistered [ 6836.461878] Key type ._llcrypt unregistered [ 6873.153442] Key type ._llcrypt registered [ 6873.156765] Key type .llcrypt registered [ 6873.270396] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 6874.435984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6874.453951] alg: No test for adler32 (adler32-zlib) [ 6875.487467] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 6875.692444] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 6877.360185] Key type lgssc registered [ 6878.392408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6885.668727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6885.682508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.982930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6887.037955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6887.204786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.576069] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6900.166050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.273662] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6900.330858] Lustre: 226390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6900.597364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6905.424805] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6913.823869] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6919.408524] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.023190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.220326] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6926.494365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6932.087055] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 6936.262504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6939.943770] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6944.089110] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6951.917391] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6961.963959] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6972.227511] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6981.018679] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7001.241470] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7001.250240] Lustre: Skipped 2 previous similar messages [ 7036.126386] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7036.134165] Lustre: Skipped 3 previous similar messages [ 7101.090546] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7101.095163] Lustre: Skipped 7 previous similar messages [ 7131.108156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7131.113157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.138556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7131.146020] Lustre: Skipped 1 previous similar message [ 7131.171752] Lustre: Skipped 1 previous similar message [ 7134.336065] Lustre: server umount lustre-OST0000 complete [ 7141.345763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7141.361474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7141.371295] Lustre: Skipped 1 previous similar message [ 7144.447693] Lustre: server umount lustre-MDT0000 complete [ 7147.924438] LustreError: 226385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871628 with bad export cookie 1615152364657475941 [ 7147.925517] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.962084] LustreError: 226385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7148.495602] Lustre: server umount lustre-MDT0001 complete [ 7160.465555] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7163.262283] Key type lgssc unregistered [ 7163.527757] LNet: 231215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7163.534204] LNetError: 231215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7164.594529] LNet: Removed LNI 192.168.203.150@tcp [ 7165.468260] Key type .llcrypt unregistered [ 7165.470315] Key type ._llcrypt unregistered [ 7190.150790] Key type ._llcrypt registered [ 7190.153228] Key type .llcrypt registered [ 7190.246770] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7191.525941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7191.595466] alg: No test for adler32 (adler32-zlib) [ 7192.702357] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 7193.065354] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7194.752594] Key type lgssc registered [ 7196.015082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.239473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7204.288324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.889597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.544471] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7215.076707] LustreError: 232638:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7215.099787] LustreError: 232638:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7219.183871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.657646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7224.180806] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7231.649335] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.440514] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.941133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7245.246709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.354803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7253.877806] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_default_debug -1 all [ 7264.582366] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.898414] Lustre: server umount lustre-OST0000 complete [ 7290.339571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.345679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.357446] Lustre: Skipped 1 previous similar message [ 7295.459165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.475261] Lustre: Skipped 2 previous similar messages [ 7298.619734] Lustre: server umount lustre-MDT0000 complete [ 7300.582434] LustreError: 232638:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7300.612693] LustreError: 232638:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7301.472156] Lustre: 231951:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787871766/real 1787871766] req@0000000032dfb9c3 x1874719328474240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787871782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7301.542550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.995162] LustreError: 235217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871783 with bad export cookie 4931424272922607337 [ 7303.004615] LustreError: MGC192.168.203.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7303.017212] LustreError: 235217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7303.583354] Lustre: server umount lustre-MDT0001 complete [ 7316.446525] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7319.538772] Key type lgssc unregistered [ 7319.965693] LNet: 236331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7319.980845] LNetError: 236331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7320.034679] LNet: Removed LNI 192.168.203.150@tcp [ 7320.968967] Key type .llcrypt unregistered [ 7320.971097] Key type ._llcrypt unregistered [ 7344.527605] Key type ._llcrypt registered [ 7344.534711] Key type .llcrypt registered [ 7344.756879] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7345.815802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7345.952331] alg: No test for adler32 (adler32-zlib) [ 7347.081807] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 7347.380217] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7349.064183] Key type lgssc registered [ 7350.246284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7622.259920] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7626.812491] Key type lgssc unregistered [ 7627.169976] LNet: 238362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7627.186889] LNetError: 238362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7627.220882] LNet: Removed LNI 192.168.203.150@tcp [ 7628.134593] Key type .llcrypt unregistered [ 7628.137915] Key type ._llcrypt unregistered [ 7657.335191] Key type ._llcrypt registered [ 7657.337759] Key type .llcrypt registered [ 7657.533527] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing set_hostid [ 7676.025244] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing load_modules_local [ 7677.370319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7677.420593] alg: No test for adler32 (adler32-zlib) [ 7678.763615] Lustre: Lustre: Build Version: 2.17.57_103_gb165317 [ 7679.072264] LNet: Added LNI 192.168.203.150@tcp [8/256/0/180] [ 7680.792161] Key type lgssc registered [ 7681.902916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7719.720215] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7467 sec ========== 19:09:58 (1787872198) [ 7721.805433] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:10:01 (1787872201) === [ 7725.457671] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:10:04 (1787872204) === [ 7770.939637] Lustre: DEBUG MARKER: oleg350-server.virtnet: executing unload_modules_local [ 7774.463275] Key type lgssc unregistered [ 7774.779415] LNet: 243609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7774.784184] LNetError: 243609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7774.817084] LNet: Removed LNI 192.168.203.150@tcp [ 7775.945948] Key type .llcrypt unregistered [ 7775.949163] Key type ._llcrypt unregistered