[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486831261 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003329] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011084] Yama: becoming mindful. [ 0.013040] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022674] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027521] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030077] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031127] *** VALIDATE tmpfs *** [ 0.033321] *** VALIDATE proc *** [ 0.034272] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038053] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044109] debug: unmapping init [mem 0xffffffffa3659000-0xffffffffa3660fff] [ 0.046175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047722] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049015] ... bit width: 48 [ 0.050017] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052021] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055284] rcu: Hierarchical SRCU implementation. [ 0.057560] smp: Bringing up secondary CPUs ... [ 0.058668] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062418] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134296] node 0 deferred pages initialised in 66ms [ 0.138126] devtmpfs: initialized [ 0.139251] x86/mm: Memory block size: 128MB [ 0.144520] gcov: version magic: 0x41383552 [ 0.148129] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151434] pinctrl core: initialized pinctrl subsystem [ 0.154305] [ 0.154967] ************************************************************* [ 0.157027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159018] ** ** [ 0.161021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163034] ** ** [ 0.166023] ** This means that this kernel is built to expose internal ** [ 0.168018] ** IOMMU data structures, which may compromise security on ** [ 0.170022] ** your system. ** [ 0.172021] ** ** [ 0.175022] ** If you see this message and you are not debugging the ** [ 0.177016] ** kernel, report this immediately to your vendor! ** [ 0.179017] ** ** [ 0.180014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182020] ************************************************************* [ 0.185812] NET: Registered protocol family 16 [ 0.187497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197052] cpuidle: using governor menu [ 0.198657] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200712] PCI: Using configuration type 1 for base access [ 0.203164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212183] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.215091] cryptd: max_cpu_qlen set to 1000 [ 0.218076] ACPI: Added _OSI(Module Device) [ 0.220018] ACPI: Added _OSI(Processor Device) [ 0.222017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228919] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236361] ACPI: Interpreter enabled [ 0.238071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239018] ACPI: Using IOAPIC for interrupt routing [ 0.242055] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253080] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268919] acpiphp: Slot [2] registered [ 0.270179] acpiphp: Slot [5] registered [ 0.271195] acpiphp: Slot [6] registered [ 0.273169] acpiphp: Slot [7] registered [ 0.275181] acpiphp: Slot [8] registered [ 0.276168] acpiphp: Slot [9] registered [ 0.278159] acpiphp: Slot [10] registered [ 0.279150] acpiphp: Slot [3] registered [ 0.281104] acpiphp: Slot [4] registered [ 0.282139] acpiphp: Slot [11] registered [ 0.284083] acpiphp: Slot [12] registered [ 0.285063] acpiphp: Slot [13] registered [ 0.285964] acpiphp: Slot [14] registered [ 0.287096] acpiphp: Slot [15] registered [ 0.288112] acpiphp: Slot [16] registered [ 0.290119] acpiphp: Slot [17] registered [ 0.292110] acpiphp: Slot [18] registered [ 0.293137] acpiphp: Slot [19] registered [ 0.294133] acpiphp: Slot [20] registered [ 0.296134] acpiphp: Slot [21] registered [ 0.297129] acpiphp: Slot [22] registered [ 0.299146] acpiphp: Slot [23] registered [ 0.301138] acpiphp: Slot [24] registered [ 0.302160] acpiphp: Slot [25] registered [ 0.304174] acpiphp: Slot [26] registered [ 0.306209] acpiphp: Slot [27] registered [ 0.307126] acpiphp: Slot [28] registered [ 0.309185] acpiphp: Slot [29] registered [ 0.311159] acpiphp: Slot [30] registered [ 0.312142] acpiphp: Slot [31] registered [ 0.314139] PCI host bridge to bus 0000:00 [ 0.315027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.318040] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.320035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333509] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337430] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347793] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353505] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368121] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370069] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373981] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.394023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.398962] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406206] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.415016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455987] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503159] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.516031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.547752] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.555023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.565026] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.581021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.591000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.600015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.608021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.640865] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.659020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.694874] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.697564] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.700457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.711107] iommu: Default domain type: Passthrough [ 0.713431] SCSI subsystem initialized [ 0.715176] ACPI: bus type USB registered [ 0.716112] usbcore: registered new interface driver usbfs [ 0.718099] usbcore: registered new interface driver hub [ 0.721095] usbcore: registered new device driver usb [ 0.723260] pps_core: LinuxPPS API ver. 1 registered [ 0.725013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729111] PTP clock support registered [ 0.731076] EDAC MC: Ver: 3.0.0 [ 0.733153] PCI: Using ACPI for IRQ routing [ 0.734704] NetLabel: Initializing [ 0.736025] NetLabel: domain hash size = 128 [ 0.738020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.740120] NetLabel: unlabeled traffic allowed by default [ 0.742268] vgaarb: loaded [ 0.744326] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.746096] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754005] clocksource: Switched to clocksource kvm-clock [ 0.869708] VFS: Disk quotas dquot_6.6.0 [ 0.871435] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.873836] *** VALIDATE ramfs *** [ 0.875188] *** VALIDATE hugetlbfs *** [ 0.876924] pnp: PnP ACPI init [ 0.879389] pnp: PnP ACPI: found 6 devices [ 0.897682] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.901279] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.903803] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.906246] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.908726] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.911528] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.914597] NET: Registered protocol family 2 [ 0.917195] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.922158] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.925930] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.931132] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.934487] TCP: Hash tables configured (established 65536 bind 65536) [ 0.937387] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.940260] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.942989] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.945804] NET: Registered protocol family 1 [ 0.948286] RPC: Registered named UNIX socket transport module. [ 0.950559] RPC: Registered udp transport module. [ 0.952204] RPC: Registered tcp transport module. [ 0.953730] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.955894] NET: Registered protocol family 44 [ 0.957464] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.959670] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.961940] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.964306] PCI: CLS 0 bytes, default 64 [ 0.966582] Unpacking initramfs... [ 2.430599] debug: unmapping init [mem 0xffffa0cdbcc54000-0xffffa0cdbffbffff] [ 2.437513] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.439825] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.444140] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.948742] Initialise system trusted keyrings [ 2.950702] Key type blacklist registered [ 2.952720] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.962828] zbud: loaded [ 2.966329] *** VALIDATE nfs *** [ 2.967615] *** VALIDATE nfs4 *** [ 2.969941] pstore: using deflate compression [ 2.973904] Platform Keyring initialized [ 3.086991] NET: Registered protocol family 38 [ 3.088894] Key type asymmetric registered [ 3.090835] Asymmetric key parser 'x509' registered [ 3.093083] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.096735] io scheduler mq-deadline registered [ 3.098703] io scheduler kyber registered [ 3.100535] io scheduler bfq registered [ 3.102702] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.106119] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.109466] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.112747] ACPI: Power Button [PWRF] [ 3.118437] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.125471] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.160754] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.168736] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.209731] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.238945] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.269210] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.280240] Non-volatile memory driver v1.3 [ 3.281956] Linux agpgart interface v0.103 [ 3.323475] virtio_blk virtio1: [vda] 147096 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.326264] vda: detected capacity change from 0 to 75313152 [ 3.346197] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.349503] vdb: detected capacity change from 0 to 1073741824 [ 3.374034] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.377249] vdc: detected capacity change from 0 to 2621440000 [ 3.407262] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.410603] vdd: detected capacity change from 0 to 2621440000 [ 3.427676] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.430887] vde: detected capacity change from 0 to 4294967296 [ 3.447801] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.450804] vdf: detected capacity change from 0 to 4294967296 [ 3.464936] libphy: Fixed MDIO Bus: probed [ 3.470445] usbcore: registered new interface driver usbserial_generic [ 3.472485] usbserial: USB Serial support registered for generic [ 3.474733] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.478182] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.480647] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.483265] mousedev: PS/2 mouse device common for all mice [ 3.486509] rtc_cmos 00:05: RTC can wake from S4 [ 3.489892] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.494152] rtc_cmos 00:05: registered as rtc0 [ 3.499477] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.502699] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.509365] intel_pstate: CPU model not supported [ 3.510241] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.515359] hid: raw HID events driver (C) Jiri Kosina [ 3.517631] usbcore: registered new interface driver usbhid [ 3.519800] usbhid: USB HID core driver [ 3.521545] drop_monitor: Initializing network drop monitor service [ 3.524087] Initializing XFRM netlink socket [ 3.526449] NET: Registered protocol family 10 [ 3.529742] Segment Routing with IPv6 [ 3.531315] NET: Registered protocol family 17 [ 3.533772] mpls_gso: MPLS GSO support [ 3.539764] RAS: Correctable Errors collector initialized. [ 3.541889] AVX version of gcm_enc/dec engaged. [ 3.543686] AES CTR mode by8 optimization enabled [ 3.634898] sched_clock: Marking stable (3634863196, 0)->(4603201050, -968337854) [ 3.638394] registered taskstats version 1 [ 3.641437] Loading compiled-in X.509 certificates [ 3.643505] zswap: loaded using pool lzo/zbud [ 3.673234] Key type big_key registered [ 3.688651] Key type encrypted registered [ 3.690501] ima: No TPM chip found, activating TPM-bypass! [ 3.692550] ima: Allocated hash algorithm: sha1 [ 3.694268] ima: No architecture policies found [ 3.696138] evm: Initialising EVM extended attributes: [ 3.697589] evm: security.selinux [ 3.698455] evm: security.ima [ 3.699195] evm: security.capability [ 3.700566] evm: HMAC attrs: 0x1 [ 3.704316] rtc_cmos 00:05: setting system clock to 2026-08-27 19:41:06 UTC (1787859666) [ 3.710905] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 3.714076] debug: unmapping init [mem 0xffffffffa3382000-0xffffffffa3658fff] [ 3.723119] Write protecting the kernel read-only data: 28672k [ 3.726534] debug: unmapping init [mem 0xffffffffa1a03000-0xffffffffa1bfffff] [ 3.729380] debug: unmapping init [mem 0xffffffffa2314000-0xffffffffa23fffff] [ 3.801672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.810177] systemd[1]: Detected virtualization kvm. [ 3.812073] systemd[1]: Detected architecture x86-64. [ 3.813872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.839493] systemd[1]: No hostname configured. [ 3.841542] systemd[1]: Set hostname to . [ 3.844553] random: systemd: uninitialized urandom read (16 bytes read) [ 3.847662] systemd[1]: Initializing machine ID from random generator. [ 3.891155] random: ln: uninitialized urandom read (6 bytes read) [ 3.985451] random: systemd: uninitialized urandom read (16 bytes read) [ 3.988601] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.992539] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.996777] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.631371] device-mapper: uevent: version 1.0.3 [ 4.633923] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.306624] virtio_net virtio0 ens2: renamed from eth0 [ 5.313603] random: fast init done [ 5.336673] scsi host0: ata_piix [ 5.354258] scsi host1: ata_piix [ 5.363553] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.367094] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.982669] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.207489] random: crng init done [ 10.209018] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.623069] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.779618] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.024802] SELinux: Disabled at runtime. [ 12.096703] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.104239] systemd[1]: Detected virtualization kvm. [ 12.105589] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.585204] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.588743] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.593415] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.597522] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.600628] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.608522] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.616591] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ 12.715874] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.075615] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.363689] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.378594] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.486352] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.501662] EDAC sbridge: Ver: 1.1.2 [ 15.213543] Key type dns_resolver registered [ 15.512175] NFS: Registering the id_resolver key type [ 15.514402] Key type id_resolver registered [ 15.516405] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg412-server login: [ 43.850851] libcfs: loading out-of-tree module taints kernel. [ 43.925675] Key type ._llcrypt registered [ 43.927606] Key type .llcrypt registered [ 44.070933] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 65.222828] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 66.933323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.961463] alg: No test for adler32 (adler32-zlib) [ 68.444414] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 69.879093] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 71.839378] Key type lgssc registered [ 74.258755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.811906] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.044364] hrtimer: interrupt took 7613000 ns [ 144.078805] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 159.519075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.548739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.853519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 160.908362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.018358] Lustre: lustre-MDT0000: new disk, initializing [ 161.139994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.170619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.414962] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.035495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.165988] Lustre: 6530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.211866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.218287] Lustre: Skipped 1 previous similar message [ 179.361277] Lustre: lustre-MDT0001: new disk, initializing [ 179.428119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.461131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.479048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.025597] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.441361] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.691691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.287088] Lustre: lustre-OST0000: new disk, initializing [ 202.299130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.319729] Lustre: 8466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.529425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.954588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.973943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.185355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.137765] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.449313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.573591] Lustre: lustre-OST0001: new disk, initializing [ 228.585027] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.594692] Lustre: 9543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.718074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.350366] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.601063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.625931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.696104] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.464402] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.724205] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.928151] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.941212] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing yml_node [ 277.408585] Lustre: DEBUG MARKER: Client: 2.17.57.124 [ 280.982711] Lustre: DEBUG MARKER: MDS: 2.17.57.124 [ 283.572803] Lustre: DEBUG MARKER: OSS: 2.17.57.124 [ 285.461368] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 15:45:46 EDT 2026 [ 301.971402] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 303.531438] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 323.041579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 323.051608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.083598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 323.562595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.577224] Lustre: Skipped 2 previous similar messages [ 326.146935] Lustre: server umount lustre-MDT0000 complete [ 331.279959] LustreError: 10358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859994 with bad export cookie 12712267767785555200 [ 331.285088] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.295851] LustreError: 10358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 333.791939] LustreError: 6538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.792713] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.793359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.793364] Lustre: Skipped 3 previous similar messages [ 333.836048] LustreError: 6538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 333.903256] Lustre: Skipped 1 previous similar message [ 336.799923] LustreError: 6536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.799979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.822176] LustreError: 6536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 336.824697] Lustre: Skipped 2 previous similar messages [ 337.773687] Lustre: server umount lustre-MDT0001 complete [ 344.229410] Lustre: server umount lustre-OST0000 complete [ 348.314029] Lustre: server umount lustre-OST0001 complete [ 356.026668] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 367.509848] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 405.195956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.614221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.646378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.755594] Lustre: lustre-MDT0000: new disk, initializing [ 405.887187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.914206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 410.373497] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 421.267147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 421.384141] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 421.433668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 421.437249] Lustre: Skipped 1 previous similar message [ 421.524807] Lustre: lustre-MDT0001: new disk, initializing [ 421.633368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.669554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.681597] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 426.141733] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 436.978399] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.973584] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.417470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.827434] Lustre: lustre-OST0000: new disk, initializing [ 450.836162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.843256] Lustre: 17069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.958509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.208885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.218278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.420618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 457.882354] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 468.014663] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 473.297404] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.624362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.479758] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.666391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.309266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.312399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.312544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.328206] Lustre: Skipped 1 previous similar message [ 483.354737] Lustre: Skipped 1 previous similar message [ 486.758902] Lustre: server umount lustre-OST0000 complete [ 493.536695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.537271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.553975] LustreError: Skipped 1 previous similar message [ 493.555171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.576885] Lustre: Skipped 1 previous similar message [ 503.779652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.798236] Lustre: Skipped 4 previous similar messages [ 505.312454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 505.781718] Lustre: server umount lustre-MDT0000 complete [ 508.925225] LustreError: 16664:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.961416] LustreError: 16664:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 510.147041] LustreError: 18609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860172 with bad export cookie 12712267767785556733 [ 510.153558] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.165435] LustreError: 18609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 510.757621] Lustre: server umount lustre-MDT0001 complete [ 515.364837] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:49:35 (1787860175) [ 522.204158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.868550] LustreError: 19336:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.046337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.982792] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 528.355047] LustreError: 19337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.398750] LustreError: 19337:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 536.854329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.276892] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 551.147886] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 557.787162] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 564.843343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.581285] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 579.725868] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 591.847067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.847584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 591.872502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.892224] Lustre: Skipped 1 previous similar message [ 596.435786] Lustre: server umount lustre-OST0000 complete [ 606.538767] Lustre: server umount lustre-MDT0000 complete [ 607.203308] LustreError: 19336:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.228586] LustreError: 19336:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 610.485186] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860273 with bad export cookie 12712267767785557615 [ 610.494840] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.501379] LustreError: 19318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 610.798553] Lustre: server umount lustre-MDT0001 complete [ 621.388682] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 624.445770] Key type lgssc unregistered [ 624.880215] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.883486] LNetError: 22871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.902339] LNet: Removed LNI 192.168.204.112@tcp [ 625.772620] Key type .llcrypt unregistered [ 625.785120] Key type ._llcrypt unregistered [ 650.462304] Key type ._llcrypt registered [ 650.469592] Key type .llcrypt registered [ 650.597755] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 651.608169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 651.741158] alg: No test for adler32 (adler32-zlib) [ 652.808966] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 653.083943] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 654.799251] Key type lgssc registered [ 656.023318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 663.395981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 663.408277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 664.786709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.711472] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 674.274764] LustreError: 24285:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.292877] LustreError: 24285:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 677.814966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.420710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 684.652398] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 693.538374] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 700.190119] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 708.323366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.779756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.830289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 715.481532] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 724.149815] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 744.931050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.955634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 744.964557] Lustre: Skipped 1 previous similar message [ 746.015860] Lustre: server umount lustre-OST0000 complete [ 750.053282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 750.053513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.056540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.080962] Lustre: Skipped 2 previous similar messages [ 755.180502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 755.183699] Lustre: Skipped 2 previous similar messages [ 760.292213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 760.299642] Lustre: Skipped 1 previous similar message [ 764.383176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 764.387038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 764.404232] Lustre: Skipped 1 previous similar message [ 764.602305] Lustre: server umount lustre-MDT0000 complete [ 768.699295] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860431 with bad export cookie 4035370834029367968 [ 768.706842] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 768.706968] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 769.122774] Lustre: server umount lustre-MDT0001 complete [ 780.307220] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 783.416774] Key type lgssc unregistered [ 783.728263] LNet: 28267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.735804] LNetError: 28267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 783.760863] LNet: Removed LNI 192.168.204.112@tcp [ 784.691834] Key type .llcrypt unregistered [ 784.698717] Key type ._llcrypt unregistered [ 813.558921] Key type ._llcrypt registered [ 813.561761] Key type .llcrypt registered [ 813.697269] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 814.849398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.883204] alg: No test for adler32 (adler32-zlib) [ 815.981952] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 816.307342] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 818.023200] Key type lgssc registered [ 819.245873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 828.243778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 828.276498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 829.834577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.588050] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 840.170678] LustreError: 29684:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 840.188913] LustreError: 29684:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 844.208583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.467381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 850.180959] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 858.336361] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 864.955780] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 878.205877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.702376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.466320] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 891.553361] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.222369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 910.817254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 910.823656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.830133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 910.872559] Lustre: Skipped 1 previous similar message [ 915.942220] LustreError: 31850:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.990507] LustreError: 31850:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 916.191598] Lustre: server umount lustre-OST0000 complete [ 926.180395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.185286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.210305] Lustre: Skipped 1 previous similar message [ 926.226171] Lustre: Skipped 2 previous similar messages [ 928.366840] Lustre: server umount lustre-MDT0000 complete [ 931.303250] LustreError: 29679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.329769] LustreError: 29679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 933.034754] LustreError: 29664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860595 with bad export cookie 6448494671237165072 [ 933.064447] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.494692] Lustre: server umount lustre-MDT0001 complete [ 945.580505] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 949.223838] Key type lgssc unregistered [ 949.620773] LNet: 33592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.630607] LNetError: 33592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.651398] LNet: Removed LNI 192.168.204.112@tcp [ 950.734165] Key type .llcrypt unregistered [ 950.737279] Key type ._llcrypt unregistered [ 976.425553] Key type ._llcrypt registered [ 976.427814] Key type .llcrypt registered [ 976.540517] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 977.982237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.136342] alg: No test for adler32 (adler32-zlib) [ 979.455986] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 979.828296] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 981.487243] Key type lgssc registered [ 982.832056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.492974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.507828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.087613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.119873] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1001.505805] LustreError: 35005:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.533114] LustreError: 35005:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1006.076241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.678526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1011.441110] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1018.706361] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.547600] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1033.035343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.465224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.545488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1040.150814] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1047.981654] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1062.375487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1062.376890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.391153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.415828] Lustre: Skipped 1 previous similar message [ 1067.491045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.494547] Lustre: Skipped 2 previous similar messages [ 1068.527766] Lustre: server umount lustre-OST0000 complete [ 1077.729430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.767235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.770870] Lustre: Skipped 1 previous similar message [ 1079.763034] Lustre: server umount lustre-MDT0000 complete [ 1082.849951] LustreError: 35002:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.869594] LustreError: 35002:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1084.097354] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860746 with bad export cookie 1377402496224712874 [ 1084.099686] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1084.106912] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1084.581353] Lustre: server umount lustre-MDT0001 complete [ 1098.156274] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1101.808065] Key type lgssc unregistered [ 1102.341925] LNet: 38564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.349099] LNetError: 38564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.383776] LNet: Removed LNI 192.168.204.112@tcp [ 1103.290445] Key type .llcrypt unregistered [ 1103.292790] Key type ._llcrypt unregistered [ 1131.403471] Key type ._llcrypt registered [ 1131.406355] Key type .llcrypt registered [ 1131.535095] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1132.612837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1132.794092] alg: No test for adler32 (adler32-zlib) [ 1133.898692] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1134.185838] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1135.879994] Key type lgssc registered [ 1137.433532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1145.662246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1145.687760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.177896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.293829] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1156.581580] LustreError: 39978:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.606735] LustreError: 39978:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1159.416850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.899393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1164.458401] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1173.050727] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1180.330508] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1187.221526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.649841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.767462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1194.471062] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1203.334958] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1211.239892] Lustre: lustre-OST0000: Not available for connect from 192.168.204.12@tcp (stopping) [ 1212.385372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.401780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.431497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.461259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1215.468990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1215.471688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.480479] Lustre: Skipped 1 previous similar message [ 1216.652323] Lustre: server umount lustre-OST0000 complete [ 1237.771285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.782131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.788927] Lustre: Skipped 1 previous similar message [ 1237.818963] Lustre: Skipped 2 previous similar messages [ 1246.178921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1246.199506] Lustre: Skipped 3 previous similar messages [ 1252.321855] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1252.437055] Lustre: server umount lustre-MDT0000 complete [ 1255.399195] LustreError: 41819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1255.424332] LustreError: 41819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1256.221062] LustreError: 39960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860919 with bad export cookie 10687199641512559675 [ 1256.230985] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.239394] LustreError: 39960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1263.034625] Lustre: server umount lustre-MDT0001 complete [ 1275.609832] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1278.733654] Key type lgssc unregistered [ 1279.142832] LNet: 43707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.156061] LNetError: 43707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.177677] LNet: Removed LNI 192.168.204.112@tcp [ 1280.174691] Key type .llcrypt unregistered [ 1280.177383] Key type ._llcrypt unregistered [ 1304.255479] Key type ._llcrypt registered [ 1304.263083] Key type .llcrypt registered [ 1304.413954] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1305.399109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1305.471659] alg: No test for adler32 (adler32-zlib) [ 1306.531941] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1306.807310] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1308.487207] Key type lgssc registered [ 1309.629652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1316.594887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1316.609798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.089932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.804045] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1327.589238] LustreError: 45120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.604431] LustreError: 45120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1330.858336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.237081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1335.954635] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1343.234381] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.614828] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.635365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.760287] LustreError: 46942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.131340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1362.171392] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1362.426308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1369.343617] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1376.135190] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.12@tcp (stopping) [ 1377.249615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1377.261762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.288605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1377.780760] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.794702] Lustre: Skipped 1 previous similar message [ 1381.207603] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.12@tcp (stopping) [ 1381.220034] Lustre: Skipped 2 previous similar messages [ 1382.443087] Lustre: server umount lustre-MDT0000 complete [ 1382.881722] LustreError: 45120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.901402] LustreError: 45120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1385.866475] LustreError: 45099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861048 with bad export cookie 5389809472510726272 [ 1385.872185] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.889049] LustreError: 45099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1386.320942] LustreError: 47560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1386.363850] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.12@tcp (stopping) [ 1388.015166] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.442547] LustreError: 47560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.443900] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.12@tcp (stopping) [ 1391.470986] LustreError: 47560:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1391.486480] Lustre: Skipped 2 previous similar messages [ 1400.287128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1400.674083] Lustre: server umount lustre-MDT0001 complete [ 1416.115557] Lustre: server umount lustre-OST0000 complete [ 1434.403731] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1436.910089] Key type lgssc unregistered [ 1437.223644] LNet: 48984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.242427] LNetError: 48984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.266760] LNet: Removed LNI 192.168.204.112@tcp [ 1438.092179] Key type .llcrypt unregistered [ 1438.094479] Key type ._llcrypt unregistered [ 1465.623425] Key type ._llcrypt registered [ 1465.629756] Key type .llcrypt registered [ 1465.756538] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1467.405159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1467.512212] alg: No test for adler32 (adler32-zlib) [ 1468.733936] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1469.000552] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1470.655423] Key type lgssc registered [ 1471.851214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1480.024537] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1480.046319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.649055] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1500.767506] LustreError: 50372:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 1527.391473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.693427] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1542.412148] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1710.583680] Lustre: server umount lustre-OST0000 complete [ 1726.556759] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1729.516580] Key type lgssc unregistered [ 1729.821196] LNet: 52051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.827863] LNetError: 52051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.845632] LNet: Removed LNI 192.168.204.112@tcp [ 1730.583777] Key type .llcrypt unregistered [ 1730.587316] Key type ._llcrypt unregistered [ 1755.622670] Key type ._llcrypt registered [ 1755.628650] Key type .llcrypt registered [ 1755.808207] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1756.973357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.039402] alg: No test for adler32 (adler32-zlib) [ 1758.083186] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1758.324489] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1760.031216] Key type lgssc registered [ 1761.188077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1770.102228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1770.136256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.867540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.279477] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1781.222170] LustreError: 53467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.257793] LustreError: 53467:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1786.344897] LustreError: 53468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.024166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.258374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1792.996259] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1802.635762] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.707924] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1815.663599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.098761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.137253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1822.316304] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1831.259752] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1836.512431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1836.517792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.527101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1836.556303] Lustre: Skipped 1 previous similar message [ 1839.658749] Lustre: server umount lustre-OST0000 complete [ 1843.682546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.695391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.710825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.722712] Lustre: Skipped 1 previous similar message [ 1846.757589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.761805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.779864] Lustre: Skipped 1 previous similar message [ 1849.758350] Lustre: server umount lustre-MDT0000 complete [ 1851.872391] LustreError: 53467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.887734] LustreError: 53467:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1853.379653] LustreError: 53448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861516 with bad export cookie 10060282302790275044 [ 1853.389214] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1853.390830] LustreError: 53448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1853.705843] Lustre: server umount lustre-MDT0001 complete [ 1864.982856] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 1869.193875] Key type lgssc unregistered [ 1869.680697] LNet: 57006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.700887] LNetError: 57006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.756756] LNet: Removed LNI 192.168.204.112@tcp [ 1871.196287] Key type .llcrypt unregistered [ 1871.201854] Key type ._llcrypt unregistered [ 1902.568981] Key type ._llcrypt registered [ 1902.575485] Key type .llcrypt registered [ 1902.849900] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 1904.482758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.552079] alg: No test for adler32 (adler32-zlib) [ 1905.897623] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 1906.214725] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 1907.983940] Key type lgssc registered [ 1909.652645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.950933] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.984277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.647427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1940.767954] LustreError: 58394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 1967.455643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.447275] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 1984.298317] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.911288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.839888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1992.927805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1998.312806] LustreError: 58419:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.356716] LustreError: 58419:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1999.386525] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2001.382198] LustreError: 59452:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.412427] LustreError: 59832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.439325] LustreError: 59832:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2007.540041] LustreError: 59430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.579837] LustreError: 59430:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2010.411744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.965856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.030777] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2024.509633] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.341684] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2033.633927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2033.644128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.656665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2036.712289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2036.713106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.716094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.513743] Lustre: server umount lustre-OST0000 complete [ 2062.306856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2062.308078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.327165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.338296] Lustre: Skipped 1 previous similar message [ 2062.358713] Lustre: Skipped 2 previous similar messages [ 2065.103256] Lustre: server umount lustre-MDT0000 complete [ 2067.424634] LustreError: 59436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.452426] LustreError: 59436:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2068.954386] LustreError: 61008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861731 with bad export cookie 7999126707749169734 [ 2068.970488] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.970667] LustreError: 61008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2083.295262] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2083.668559] Lustre: server umount lustre-MDT0001 complete [ 2097.570980] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2100.983900] Key type lgssc unregistered [ 2101.370637] LNet: 62144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.384597] LNetError: 62144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.419894] LNet: Removed LNI 192.168.204.112@tcp [ 2102.399545] Key type .llcrypt unregistered [ 2102.403219] Key type ._llcrypt unregistered [ 2128.131252] Key type ._llcrypt registered [ 2128.134550] Key type .llcrypt registered [ 2128.274944] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2129.457461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.562260] alg: No test for adler32 (adler32-zlib) [ 2130.581717] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2130.853850] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2132.511181] Key type lgssc registered [ 2134.410607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2142.675770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2142.713690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.268628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.927868] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2154.466720] LustreError: 63559:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.492283] LustreError: 63559:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2157.919252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.312268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.416910] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2171.370580] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2176.683842] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.998344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.562464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.396242] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2189.701380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2200.081397] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2218.467100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.469440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.491558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.506951] Lustre: Skipped 1 previous similar message [ 2218.518335] Lustre: Skipped 1 previous similar message [ 2223.384455] Lustre: server umount lustre-OST0000 complete [ 2228.717902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.721472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.740393] Lustre: Skipped 1 previous similar message [ 2228.762231] Lustre: Skipped 1 previous similar message [ 2233.412643] Lustre: server umount lustre-MDT0000 complete [ 2233.824822] LustreError: 63559:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.842324] LustreError: 63559:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2237.419259] LustreError: 63539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861900 with bad export cookie 5765336155959398666 [ 2237.422773] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2237.432029] LustreError: 63539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2237.750087] Lustre: server umount lustre-MDT0001 complete [ 2249.238848] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2251.761685] Key type lgssc unregistered [ 2252.003550] LNet: 67117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.010796] LNetError: 67117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.025784] LNet: Removed LNI 192.168.204.112@tcp [ 2252.631535] Key type .llcrypt unregistered [ 2252.633629] Key type ._llcrypt unregistered [ 2277.679728] Key type ._llcrypt registered [ 2277.682464] Key type .llcrypt registered [ 2277.774799] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2278.898647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2279.048659] alg: No test for adler32 (adler32-zlib) [ 2280.026683] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2280.253964] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2281.890677] Key type lgssc registered [ 2283.264602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.077917] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:19:15 (1787861955) [ 2304.609123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2304.631293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.690222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.195624] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2316.258586] LustreError: 68824:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.287217] LustreError: 68824:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2321.379478] LustreError: 68823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.493642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.008078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2330.900629] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2339.660635] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2345.529233] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2353.188257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.508888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.452930] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2369.648552] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.048838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2386.775248] Lustre: Failing over lustre-MDT0000 [ 2386.989636] Lustre: server umount lustre-MDT0000 complete [ 2387.426417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.428497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.428962] LustreError: 70560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.438320] Lustre: Skipped 1 previous similar message [ 2391.951372] LustreError: 68818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.991251] LustreError: 68818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2397.010699] LustreError: 68820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.084914] LustreError: 68820:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2398.509358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.817762] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.070826] Lustre: *** cfs_fail_loc=135, val=0*** [ 2399.227625] LustreError: 71938:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2399.240673] LustreError: 71938:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -2 [ 2399.265415] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2399.265415] [ 2399.285966] LustreError: MGC192.168.204.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2399.312803] LustreError: 71926:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2399.326687] LustreError: 71926:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2399.333504] LustreError: 71926:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2399.342601] LustreError: 71926:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2399.453319] Lustre: server umount lustre-MDT0000 complete [ 2399.461349] LustreError: 71926:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2407.106361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.250682] LustreError: 68820:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.291963] LustreError: 68820:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2407.331712] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.338514] LustreError: 68815:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2407.601318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.685613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2408.336278] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2413.026312] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2413.037072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2413.070262] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2413.152735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2423.272212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2423.274830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.303771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2423.326919] Lustre: Skipped 2 previous similar messages [ 2423.339614] Lustre: Skipped 1 previous similar message [ 2427.805823] Lustre: server umount lustre-OST0000 complete [ 2438.627301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.648992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.662954] Lustre: Skipped 1 previous similar message [ 2440.938759] Lustre: server umount lustre-MDT0000 complete [ 2443.760136] LustreError: 72766:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.809245] LustreError: 72766:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2446.717514] LustreError: 71428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862109 with bad export cookie 15502739988775570555 [ 2446.729218] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.741046] LustreError: 71428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2447.799827] Lustre: server umount lustre-MDT0001 complete [ 2463.008534] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2466.050873] Key type lgssc unregistered [ 2466.419123] LNet: 73723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.434902] LNetError: 73723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.455765] LNet: Removed LNI 192.168.204.112@tcp [ 2467.427136] Key type .llcrypt unregistered [ 2467.430362] Key type ._llcrypt unregistered [ 2498.905710] Key type ._llcrypt registered [ 2498.915980] Key type .llcrypt registered [ 2499.049682] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2500.571858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.610277] alg: No test for adler32 (adler32-zlib) [ 2502.335530] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2502.565934] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2504.295348] Key type lgssc registered [ 2505.806546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2516.296569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.340185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.047428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.764720] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2528.225792] LustreError: 75137:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.255921] LustreError: 75137:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2531.130652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.517208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.637642] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2543.466356] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2548.516306] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2554.483364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2554.880172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.005473] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2564.152044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2569.930923] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2576.406137] Lustre: Failing over lustre-MDT0000 [ 2576.745172] Lustre: server umount lustre-MDT0000 complete [ 2577.378587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.387981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.403902] LustreError: 75136:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.427127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.427865] LustreError: 75864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2579.446366] Lustre: Skipped 1 previous similar message [ 2579.464997] LustreError: 75864:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2582.865346] LustreError: 76426:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.891478] LustreError: 76426:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2587.984163] LustreError: 77079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.018104] LustreError: 77079:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2588.550165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.745710] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2588.946645] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2589.100939] LustreError: 78223:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2589.107799] LustreError: 78223:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -9 [ 2589.120119] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2589.120119] [ 2589.136403] LustreError: MGC192.168.204.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2589.163515] LustreError: 78211:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2589.175038] LustreError: 78211:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2589.181113] LustreError: 78211:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2589.189831] LustreError: 78211:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2589.240876] Lustre: server umount lustre-MDT0000 complete [ 2589.247875] LustreError: 78211:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2596.866094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.019983] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2597.031338] LustreError: 75129:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2597.043463] LustreError: 75133:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.079961] LustreError: 75133:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 2597.379381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2597.480717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2598.223351] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2602.481689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2602.516497] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2602.575375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2603.085966] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2612.720766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.738256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.742754] Lustre: Skipped 1 previous similar message [ 2612.761452] Lustre: Skipped 1 previous similar message [ 2617.826016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2618.115636] Lustre: server umount lustre-OST0000 complete [ 2622.946146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.947837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2622.958482] Lustre: Skipped 1 previous similar message [ 2622.958929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.980172] Lustre: Skipped 2 previous similar messages [ 2628.068456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.077764] Lustre: Skipped 1 previous similar message [ 2628.769617] Lustre: server umount lustre-MDT0000 complete [ 2633.184617] LustreError: 75137:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.214125] LustreError: 75137:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2633.288420] LustreError: 75118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862296 with bad export cookie 17106233995509497063 [ 2633.301144] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.305672] LustreError: 75118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2633.650453] Lustre: server umount lustre-MDT0001 complete [ 2648.474227] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2651.718307] Key type lgssc unregistered [ 2652.093189] LNet: 80010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.103589] LNetError: 80010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2653.164212] LNet: Removed LNI 192.168.204.112@tcp [ 2654.261891] Key type .llcrypt unregistered [ 2654.267410] Key type ._llcrypt unregistered [ 2669.397890] Key type ._llcrypt registered [ 2669.400297] Key type .llcrypt registered [ 2669.510926] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2669.890368] Key type .llcrypt unregistered [ 2669.900588] Key type ._llcrypt unregistered [ 2701.444665] Key type ._llcrypt registered [ 2701.455473] Key type .llcrypt registered [ 2701.626980] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2702.485843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2702.598124] alg: No test for adler32 (adler32-zlib) [ 2704.178954] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2704.562708] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2706.268285] Key type lgssc registered [ 2707.830204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2715.373612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2715.390928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2716.944446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.556736] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2726.375313] LustreError: 81925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.393271] LustreError: 81925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2731.488745] LustreError: 81926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.081608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.786879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2739.885810] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2747.982152] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2754.894380] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2762.839531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.421668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.104347] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2779.300632] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2787.006398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2799.584857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.593030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.607071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.661907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.670702] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.686826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.014776] Lustre: server umount lustre-OST0000 complete [ 2810.854641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2810.863912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.881197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.886243] Lustre: Skipped 1 previous similar message [ 2812.896810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.900111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.354493] Lustre: server umount lustre-MDT0000 complete [ 2818.023316] LustreError: 81921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.049488] LustreError: 81921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2819.638273] LustreError: 81905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862482 with bad export cookie 1993192254343209147 [ 2819.654146] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.667936] LustreError: 81905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2820.192422] Lustre: server umount lustre-MDT0001 complete [ 2830.995110] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2834.169851] Key type lgssc unregistered [ 2834.479076] LNet: 85484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2834.484990] LNetError: 85484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.558507] LNet: Removed LNI 192.168.204.112@tcp [ 2836.366093] Key type .llcrypt unregistered [ 2836.367907] Key type ._llcrypt unregistered [ 2860.014614] Key type ._llcrypt registered [ 2860.016796] Key type .llcrypt registered [ 2860.101700] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 2861.259184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2861.292568] alg: No test for adler32 (adler32-zlib) [ 2862.346607] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 2862.605374] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 2864.327226] Key type lgssc registered [ 2865.568459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.864665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.879376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.451145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.112607] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2885.606421] LustreError: 86899:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.615067] LustreError: 86899:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2889.354844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.920891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.502844] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2902.063610] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2907.973820] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2915.096219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2915.653238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.756593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2921.722716] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 2929.723712] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2940.386378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.389602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.392917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.413757] Lustre: Skipped 1 previous similar message [ 2942.171905] Lustre: server umount lustre-OST0000 complete [ 2946.017299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.029044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.048863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.055753] Lustre: Skipped 1 previous similar message [ 2950.625654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.629124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.753656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.763688] Lustre: Skipped 2 previous similar messages [ 2960.352549] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.864413] LustreError: 87627:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.880614] LustreError: 87627:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2962.599330] Lustre: server umount lustre-MDT0000 complete [ 2965.989326] LustreError: 86900:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.019439] LustreError: 86900:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2966.444434] LustreError: 86881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862629 with bad export cookie 17683878451518913225 [ 2966.448726] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.457814] LustreError: 86881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2966.760686] Lustre: server umount lustre-MDT0001 complete [ 2978.436615] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 2981.353392] Key type lgssc unregistered [ 2981.746589] LNet: 90438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2981.756795] LNetError: 90438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2981.782909] LNet: Removed LNI 192.168.204.112@tcp [ 2982.837459] Key type .llcrypt unregistered [ 2982.840538] Key type ._llcrypt unregistered [ 3008.549837] Key type ._llcrypt registered [ 3008.551940] Key type .llcrypt registered [ 3008.636816] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3009.596638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3009.616696] alg: No test for adler32 (adler32-zlib) [ 3010.641342] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3010.840726] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3012.527204] Key type lgssc registered [ 3013.575045] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3021.578750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3021.603509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.069426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.754866] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3032.554522] LustreError: 91853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3032.573982] LustreError: 91853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3036.877981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.462826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.224251] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3050.482566] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3056.456977] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3065.919483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.484413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.536386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3073.344895] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3080.216864] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3096.043601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3096.064269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.090958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3096.887182] Lustre: server umount lustre-OST0000 complete [ 3103.716587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.740809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.746336] Lustre: Skipped 1 previous similar message [ 3103.790864] Lustre: Skipped 1 previous similar message [ 3107.388591] Lustre: server umount lustre-MDT0000 complete [ 3108.832366] LustreError: 91853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.857556] LustreError: 91853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3112.625823] LustreError: 91833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787862775 with bad export cookie 16227553847826173249 [ 3112.648597] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.331422] Lustre: server umount lustre-MDT0001 complete [ 3134.294962] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3137.519971] Key type lgssc unregistered [ 3137.823917] LNet: 95412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3137.831557] LNetError: 95412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3137.849729] LNet: Removed LNI 192.168.204.112@tcp [ 3138.707712] Key type .llcrypt unregistered [ 3138.709773] Key type ._llcrypt unregistered [ 3163.113848] Key type ._llcrypt registered [ 3163.116018] Key type .llcrypt registered [ 3163.228817] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3164.474051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3164.535784] alg: No test for adler32 (adler32-zlib) [ 3165.567881] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3165.820374] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3167.559174] Key type lgssc registered [ 3168.946464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3176.720809] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3176.740876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.287569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3197.407444] LustreError: 96797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 3224.159299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3231.653584] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3240.314807] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.538771] Lustre: server umount lustre-OST0000 complete [ 3257.886551] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:35:18 (1787862918) [ 3265.104782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.634185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.012781] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3270.624466] LustreError: 98438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.646861] LustreError: 98438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3275.748832] LustreError: 98439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.019906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.594907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3282.912615] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3291.262708] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3297.545126] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3304.611082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.990041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.646147] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3311.179547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3318.970182] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3333.604317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3333.608258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.626982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.633411] Lustre: Skipped 1 previous similar message [ 3338.727742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.736119] Lustre: Skipped 2 previous similar messages [ 3338.940370] Lustre: server umount lustre-OST0000 complete [ 3343.841074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.846574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.858021] Lustre: Skipped 1 previous similar message [ 3343.871867] Lustre: Skipped 1 previous similar message [ 3348.549908] Lustre: server umount lustre-MDT0000 complete [ 3348.962530] LustreError: 99703:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.996439] LustreError: 99703:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3352.598491] LustreError: 98420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863015 with bad export cookie 9225935225363446784 [ 3352.603213] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.611783] LustreError: 98420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3353.015968] Lustre: server umount lustre-MDT0001 complete [ 3364.502996] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3367.407825] Key type lgssc unregistered [ 3367.729957] LNet: 102133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.747553] LNetError: 102133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.764609] LNet: Removed LNI 192.168.204.112@tcp [ 3368.709875] Key type .llcrypt unregistered [ 3368.712122] Key type ._llcrypt unregistered [ 3413.574805] Key type ._llcrypt registered [ 3413.580726] Key type .llcrypt registered [ 3413.714478] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3415.244860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.306405] alg: No test for adler32 (adler32-zlib) [ 3416.572295] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3416.961336] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3418.695176] Key type lgssc registered [ 3419.800617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3429.803407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3429.844121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.368202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3435.891270] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3441.637586] LustreError: 104320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.676532] LustreError: 104320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3446.016408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.354056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.030331] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3464.292749] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3471.673353] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3480.005426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.457590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.066627] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3495.534779] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3502.008722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3512.300544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.305202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3512.318121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.325588] LustreError: Skipped 1 previous similar message [ 3512.341822] Lustre: Skipped 1 previous similar message [ 3517.414193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.427025] Lustre: Skipped 1 previous similar message [ 3517.830814] Lustre: server umount lustre-OST0000 complete [ 3523.040929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3523.048032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.060362] Lustre: Skipped 1 previous similar message [ 3523.065286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3527.667254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.670502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3527.690265] Lustre: Skipped 1 previous similar message [ 3532.770161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3532.778616] Lustre: Skipped 1 previous similar message [ 3537.376348] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3537.526632] Lustre: server umount lustre-MDT0000 complete [ 3537.904161] LustreError: 104321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.942111] LustreError: 104321:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3542.153165] LustreError: 104301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863204 with bad export cookie 289930826385682165 [ 3542.164730] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.168703] LustreError: 104301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3542.773090] Lustre: server umount lustre-MDT0001 complete [ 3554.077340] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 3556.642889] Key type lgssc unregistered [ 3556.941762] LNet: 107880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3556.952214] LNetError: 107880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3556.977325] LNet: Removed LNI 192.168.204.112@tcp [ 3557.900120] Key type .llcrypt unregistered [ 3557.903757] Key type ._llcrypt unregistered [ 3577.018320] Key type ._llcrypt registered [ 3577.021651] Key type .llcrypt registered [ 3577.140762] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3578.343062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3578.403180] alg: No test for adler32 (adler32-zlib) [ 3579.643030] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 3579.922337] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 3581.609357] Key type lgssc registered [ 3583.142917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3590.767494] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3590.790551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.368531] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3611.487727] LustreError: 109124:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 3638.175145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.593863] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3654.167527] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3661.441632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3661.769826] LustreError: MGC192.168.204.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3661.789137] LustreError: 110149:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3661.803537] LustreError: 110149:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3661.814944] LustreError: 110149:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3661.908391] Lustre: server umount lustre-MDT0000 complete [ 3661.915896] LustreError: 110149:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3681.247295] Lustre: 108460:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787863328/real 1787863328] req@0000000029702973 x1874710486846848/t0(0) o400->MGC192.168.204.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787863344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3681.284340] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.309302] LustreError: 109137:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3701.906537] Lustre: server umount lustre-OST0000 complete [ 3711.288179] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 3722.579750] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3759.500050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.767680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3759.805326] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3759.875949] Lustre: lustre-MDT0000: new disk, initializing [ 3760.044403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3760.085773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3764.502500] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3775.443492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3775.560894] Lustre: 112945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3775.621907] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3775.634989] Lustre: Skipped 1 previous similar message [ 3775.726624] Lustre: lustre-MDT0001: new disk, initializing [ 3775.852651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3775.882780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3775.891819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3780.440953] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3788.921448] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3794.140082] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3801.305037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3801.612502] Lustre: lustre-OST0000: new disk, initializing [ 3801.615836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3801.619568] Lustre: 114914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3801.714145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.242082] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3811.939506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3811.956571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3812.104780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3820.904571] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3827.363552] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3827.628531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3832.706484] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3833.065249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3837.934644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.937362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3837.960516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3837.961560] LustreError: Skipped 1 previous similar message [ 3837.984197] Lustre: Skipped 1 previous similar message [ 3841.589568] Lustre: server umount lustre-OST0000 complete [ 3847.648384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3847.659028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3847.672707] Lustre: Skipped 1 previous similar message [ 3847.687270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3852.217350] Lustre: server umount lustre-MDT0000 complete [ 3855.677081] LustreError: 112938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863518 with bad export cookie 13397495028692000855 [ 3855.693889] LustreError: 112938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3855.693908] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3856.001551] Lustre: server umount lustre-MDT0001 complete [ 3863.886880] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:45:24 (1787863524) [ 3867.409945] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3882.042526] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 3890.619333] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 3925.072235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.355335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3925.394670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3925.498962] Lustre: lustre-MDT0000: new disk, initializing [ 3925.619137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3925.645636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3930.029392] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3939.666763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.790995] Lustre: 119827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3939.860516] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3939.863556] Lustre: Skipped 1 previous similar message [ 3939.924977] Lustre: lustre-MDT0001: new disk, initializing [ 3940.014894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.058893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3940.071557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3943.788071] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3952.038634] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.519634] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.107967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.388107] Lustre: lustre-OST0000: new disk, initializing [ 3963.411616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.418950] Lustre: 121792:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3963.529431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.213995] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 3970.638925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3970.652991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3970.812822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3977.748679] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3981.630985] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3981.800756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3985.573365] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3985.773886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3991.520179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.521462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.527631] LustreError: Skipped 1 previous similar message [ 3991.531607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.535327] Lustre: Skipped 2 previous similar messages [ 3991.556675] Lustre: Skipped 3 previous similar messages [ 3993.655775] Lustre: server umount lustre-OST0000 complete [ 4001.761023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.763398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.768885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.804098] Lustre: Skipped 1 previous similar message [ 4003.162474] Lustre: server umount lustre-MDT0000 complete [ 4006.882131] LustreError: 119840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.904053] LustreError: 119840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4006.927129] LustreError: 119820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863669 with bad export cookie 13397495028692001737 [ 4006.928861] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4006.933162] LustreError: 119820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4007.376269] Lustre: server umount lustre-MDT0001 complete [ 4015.117697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.472459] LustreError: 124010:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4015.554738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.104586] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4020.709351] LustreError: 124011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4025.833207] LustreError: 124010:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4027.137507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.995484] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4038.500525] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4043.621696] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4050.039322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4056.341427] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4063.451726] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4070.706532] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4076.009251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.010349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4076.029165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.029187] Lustre: Skipped 1 previous similar message [ 4081.582470] Lustre: server umount lustre-OST0000 complete [ 4088.801331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4088.808478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.823554] Lustre: Skipped 1 previous similar message [ 4088.833530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.846897] Lustre: Skipped 3 previous similar messages [ 4092.032143] Lustre: server umount lustre-MDT0000 complete [ 4094.433027] LustreError: 124006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4094.447047] LustreError: 124006:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4095.670447] LustreError: 123991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863758 with bad export cookie 13397495028692002619 [ 4095.674393] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.691517] LustreError: 123991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4096.165271] Lustre: server umount lustre-MDT0001 complete [ 4108.853337] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 4111.545855] Key type lgssc unregistered [ 4111.849496] LNet: 127643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4111.867564] LNetError: 127643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4111.921443] LNet: Removed LNI 192.168.204.112@tcp [ 4112.816346] Key type .llcrypt unregistered [ 4112.821845] Key type ._llcrypt unregistered [ 4129.137538] Key type ._llcrypt registered [ 4129.140831] Key type .llcrypt registered [ 4129.247311] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 4141.703756] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 4142.535213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.663457] alg: No test for adler32 (adler32-zlib) [ 4143.679183] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 4143.987798] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 4145.655205] Key type lgssc registered [ 4146.818074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4181.860023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4181.908361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.413612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4183.537772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4183.676297] Lustre: lustre-MDT0000: new disk, initializing [ 4183.844783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.882498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4188.777815] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4200.360045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.514936] Lustre: 130792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4200.580947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4200.585859] Lustre: Skipped 1 previous similar message [ 4200.674300] Lustre: lustre-MDT0001: new disk, initializing [ 4200.822204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4200.911572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4200.926962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4205.523794] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4215.533369] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4221.517594] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4229.616276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.195733] Lustre: lustre-OST0000: new disk, initializing [ 4230.199909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4230.212874] Lustre: 132760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4230.418817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.939181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4235.952385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4236.077584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4237.808825] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4248.840338] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4254.007122] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4254.242078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4260.350791] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4260.738768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4266.979501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.992495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4267.027231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4267.032742] Lustre: Skipped 1 previous similar message [ 4272.099375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.119166] Lustre: Skipped 1 previous similar message [ 4277.215506] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4277.223639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.248048] Lustre: Skipped 1 previous similar message [ 4277.491442] Lustre: server umount lustre-OST0000 complete [ 4282.339509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.343696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.367225] Lustre: Skipped 2 previous similar messages [ 4282.377751] Lustre: Skipped 1 previous similar message [ 4287.458638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.471615] Lustre: Skipped 1 previous similar message [ 4287.621511] Lustre: server umount lustre-MDT0000 complete [ 4292.544627] LustreError: 132759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787863955 with bad export cookie 2597232371126391492 [ 4292.546133] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.563375] LustreError: 132759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4293.065891] Lustre: server umount lustre-MDT0001 complete [ 4303.723868] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 16:52:44 (1787863964) [ 4311.494372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.386803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.211235] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4317.665811] LustreError: 135176:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4317.689777] LustreError: 135176:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4322.791659] LustreError: 135177:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.851666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.250319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.209698] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4340.391493] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4346.666313] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4351.079174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.099832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4351.108594] Lustre: Skipped 1 previous similar message [ 4355.478218] Lustre: server umount lustre-MDT0000 complete [ 4356.591733] LustreError: 135177:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.612785] LustreError: 135177:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4359.252400] LustreError: 135158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864022 with bad export cookie 2597232371126392374 [ 4359.257032] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4359.271684] LustreError: 135158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4359.633262] Lustre: server umount lustre-MDT0001 complete [ 4367.271453] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 16:53:48 (1787864028) [ 4372.609248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4388.256071] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4393.377110] LustreError: 137589:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 4419.039460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.761594] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4433.139354] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4435.212540] Lustre: server umount lustre-OST0000 complete [ 4443.106849] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 16:55:04 (1787864104) [ 4450.298062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.716438] LustreError: 139037:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.775821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.311598] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4455.912840] LustreError: 139038:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.869395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.031244] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4477.169731] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4483.294897] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.014895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.155745] LustreError: 140854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.185381] LustreError: 140854:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4490.351897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.357772] Lustre: Skipped 1 previous similar message [ 4496.355686] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4503.714457] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4510.689160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4510.700461] LustreError: Skipped 1 previous similar message [ 4510.706264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.726991] Lustre: Skipped 1 previous similar message [ 4510.733167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.776406] Lustre: server umount lustre-MDT0000 complete [ 4520.339048] LustreError: 139017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864183 with bad export cookie 2597232371126393039 [ 4520.352877] LustreError: 139017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4520.357130] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.769774] Lustre: server umount lustre-MDT0001 complete [ 4534.846710] Lustre: server umount lustre-OST0000 complete [ 4543.501468] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 16:56:44 (1787864204) [ 4549.731162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.126899] LustreError: 142689:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.247302] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4562.589471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.018428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4563.033837] Lustre: Skipped 1 previous similar message [ 4567.768457] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4574.464806] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4579.548452] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.032165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4592.440783] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4595.586621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4599.113813] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4603.387181] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4603.582160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4607.517574] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4607.667481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4611.061350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.063785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4611.076902] Lustre: Skipped 2 previous similar messages [ 4611.078715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.098223] Lustre: Skipped 7 previous similar messages [ 4615.393261] Lustre: server umount lustre-OST0000 complete [ 4619.232935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4619.237606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.244814] Lustre: Skipped 1 previous similar message [ 4624.727827] Lustre: server umount lustre-MDT0000 complete [ 4626.401512] LustreError: 142690:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.417179] LustreError: 142690:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4628.079968] LustreError: 142671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864290 with bad export cookie 2597232371126394110 [ 4628.084989] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4628.091689] LustreError: 142671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4628.360790] Lustre: server umount lustre-MDT0001 complete [ 4638.495319] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 16:58:19 (1787864299) [ 4644.575913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.063550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4665.183510] LustreError: 146849:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 4696.204306] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4702.741137] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4708.524209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.980860] LustreError: 146874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.046865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.051862] Lustre: Skipped 2 previous similar messages [ 4709.109052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4712.746624] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4720.389940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.372243] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4730.499910] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4735.764828] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4739.640934] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4739.834211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4743.820955] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4744.035428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4746.213378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4746.217904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.224850] Lustre: Skipped 1 previous similar message [ 4746.227958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.232884] Lustre: Skipped 3 previous similar messages [ 4751.877908] Lustre: server umount lustre-MDT0000 complete [ 4754.828569] LustreError: 146857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864417 with bad export cookie 2597232371126395132 [ 4754.831943] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4754.837838] LustreError: 146857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4755.059813] Lustre: server umount lustre-MDT0001 complete [ 4768.750203] Lustre: server umount lustre-OST0000 complete [ 4776.594617] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:00:37 (1787864437) [ 4782.935573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.496954] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4803.615467] LustreError: 151040:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 4835.561601] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4842.726545] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4849.781023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.879988] LustreError: 129012:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e42fe561 x1874711078394624/t0(0) o250->MGC192.168.204.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4850.114580] LustreError: 151065:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.131072] LustreError: 151065:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4850.277099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4854.822763] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4862.562991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4867.118820] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4874.399765] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4880.678968] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4886.956136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.134510] Lustre: lustre-OST0001: new disk, initializing [ 4887.144469] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4887.152976] Lustre: 153923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4894.344353] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 4896.304846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4896.310553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4896.459308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4904.529845] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.228915] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4909.503523] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4914.764819] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4914.970808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4917.238194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.243626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.262643] Lustre: Skipped 3 previous similar messages [ 4917.279588] Lustre: Skipped 4 previous similar messages [ 4923.266985] Lustre: server umount lustre-OST0000 complete [ 4927.456588] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.744826] Lustre: server umount lustre-OST0001 complete [ 4945.240896] Lustre: server umount lustre-MDT0000 complete [ 4948.897892] LustreError: 151048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864611 with bad export cookie 2597232371126396084 [ 4948.903821] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.909076] LustreError: 151048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4949.406978] Lustre: server umount lustre-MDT0001 complete [ 4974.432558] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:03:55 (1787864635) [ 4976.252245] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4978.122965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4979.743778] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:04:00 (1787864640) [ 4987.025330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.308892] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4987.365707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4987.552359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4987.558596] Lustre: Skipped 5 previous similar messages [ 4992.985227] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5003.057913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.137995] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5003.175207] Lustre: 157486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5008.064255] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5017.222339] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5024.430278] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5031.618913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5031.849485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5038.267866] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5038.812089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5045.483028] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5050.151597] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5050.349271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5054.934396] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5055.091666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5059.553917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.559045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.564128] LustreError: Skipped 2 previous similar messages [ 5059.572937] Lustre: Skipped 4 previous similar messages [ 5063.640997] Lustre: server umount lustre-OST0000 complete [ 5085.310745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.746695] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5098.800576] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5100.039487] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.050547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.060267] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5100.085286] Lustre: Skipped 1 previous similar message [ 5103.499520] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5103.695839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5107.661279] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5107.924451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5113.326453] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ac952824000.ost_server_uuid 50 [ 5114.745822] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ac952824000.ost_server_uuid in FULL state after 0 sec [ 5114.912278] LustreError: 157493:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.12@tcp) returned error from blocking AST (req@000000009c958859 x1874711078498048 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000009c90197e/0x240b3914b0226e0e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.12@tcp remote: 0xef3f47d02f8cf091 expref: 5 pid: 157492 timeout: 5214 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5114.958108] LustreError: lustre-MDT0000: A client on nid 192.168.204.12@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5114.982093] LustreError: 157482:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.12@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000009c90197e/0x240b3914b0226e0e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.12@tcp remote: 0xef3f47d02f8cf091 expref: 6 pid: 157492 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5120.482024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.493429] LustreError: Skipped 1 previous similar message [ 5125.580602] Lustre: server umount lustre-OST0000 complete [ 5136.086656] Lustre: server umount lustre-MDT0000 complete [ 5140.372887] LustreError: 157478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864803 with bad export cookie 2597232371126397267 [ 5140.381896] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.387250] LustreError: 157478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5146.888982] Lustre: server umount lustre-MDT0001 complete [ 5158.680376] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5161.557970] Key type lgssc unregistered [ 5161.883214] LNet: 163610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.902832] LNetError: 163610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.935497] LNet: Removed LNI 192.168.204.112@tcp [ 5162.613085] Key type .llcrypt unregistered [ 5162.615822] Key type ._llcrypt unregistered [ 5184.231449] Key type ._llcrypt registered [ 5184.235700] Key type .llcrypt registered [ 5184.429564] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5185.389728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5185.515763] alg: No test for adler32 (adler32-zlib) [ 5186.636746] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5186.998586] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5188.719172] Key type lgssc registered [ 5189.840836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5197.811980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5197.833879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.246636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.965919] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5209.575128] LustreError: 165003:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.597529] LustreError: 165003:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5213.577833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.007696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5219.158557] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5229.087659] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5236.199233] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.939290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.428582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5245.438022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5249.812739] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5256.797556] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5262.853342] Lustre: Failing over lustre-MDT0000 [ 5263.124291] Lustre: server umount lustre-MDT0000 complete [ 5263.842044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5263.849143] LustreError: 165730:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.853121] Lustre: Skipped 1 previous similar message [ 5263.879677] LustreError: 165730:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5268.971965] LustreError: 164997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.008551] LustreError: 164997:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5274.082303] LustreError: 165002:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.097770] LustreError: 165002:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5275.132755] LustreError: 167584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.12@tcp arrived at 1787864937 with bad export cookie 16110367314645890399 [ 5275.161686] LustreError: 167584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5279.203888] LustreError: 167447:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.208091] Lustre: 164316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787864926/real 1787864926] req@000000002e45fc0a x1874712171931776/t0(0) o400->MGC192.168.204.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787864942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5279.230494] LustreError: 167447:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5279.264529] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5281.977515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.448589] LustreError: 165003:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.464480] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdf9385b75e8835c8 [ 5289.492817] LustreError: 165003:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5289.512580] Lustre: MGC192.168.204.112@tcp: Connection restored to 0@lo (at 0@lo) [ 5289.792951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.849846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5294.363183] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5295.073939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5295.091647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5296.852712] Lustre: lustre-MDT0000: Denying connection for new client f9bb73ee-4b9e-4ed7-a4e8-5f128c496ec5 (at 192.168.204.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5305.319132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.323731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5305.337517] Lustre: Skipped 2 previous similar messages [ 5310.436251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.454036] Lustre: Skipped 2 previous similar messages [ 5311.694663] Lustre: server umount lustre-OST0000 complete [ 5315.623864] LustreError: 168737:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5315.632524] Lustre: 168085:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5315.645242] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5315.658916] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5315.667338] LustreError: 168085:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008dc5a9eb x1874712171951104/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5315.682687] LustreError: 168085:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5315.689208] LustreError: 168085:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5315.694953] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5316.005872] Lustre: server umount lustre-MDT0000 complete [ 5319.876370] LustreError: 167584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787864982 with bad export cookie 16110367314645890504 [ 5319.898456] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5319.914389] LustreError: 164312:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5322.276240] Lustre: server umount lustre-MDT0001 complete [ 5336.237090] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5339.049747] Key type lgssc unregistered [ 5339.382070] LNet: 169492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5339.393584] LNetError: 169492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5339.414302] LNet: Removed LNI 192.168.204.112@tcp [ 5340.165630] Key type .llcrypt unregistered [ 5340.184265] Key type ._llcrypt unregistered [ 5361.245841] Key type ._llcrypt registered [ 5361.252312] Key type .llcrypt registered [ 5361.428126] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5362.554089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5362.679127] alg: No test for adler32 (adler32-zlib) [ 5363.726200] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5363.913070] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5365.608350] Key type lgssc registered [ 5366.619322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5374.197674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5374.213941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5375.745919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.128679] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5386.209400] LustreError: 170884:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.230942] LustreError: 170884:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5387.463928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.780227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5391.648512] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5397.974495] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5402.958894] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5409.615649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5410.110229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.571321] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5421.449326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5423.203411] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5431.778593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.799617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5431.800051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5436.638859] Lustre: server umount lustre-OST0000 complete [ 5442.024507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.044097] Lustre: Skipped 1 previous similar message [ 5442.049390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.054820] Lustre: Skipped 2 previous similar messages [ 5447.145955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.150869] Lustre: Skipped 1 previous similar message [ 5447.472233] Lustre: server umount lustre-MDT0000 complete [ 5450.806154] LustreError: 170864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865113 with bad export cookie 240207926584301319 [ 5450.808875] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5450.819596] LustreError: 170864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5451.099166] Lustre: server umount lustre-MDT0001 complete [ 5463.059306] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5465.868774] Key type lgssc unregistered [ 5466.207172] LNet: 174421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5466.228881] LNetError: 174421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5466.242974] LNet: Removed LNI 192.168.204.112@tcp [ 5467.309995] Key type .llcrypt unregistered [ 5467.312785] Key type ._llcrypt unregistered [ 5497.287717] Key type ._llcrypt registered [ 5497.290438] Key type .llcrypt registered [ 5497.386641] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5498.296666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.332120] alg: No test for adler32 (adler32-zlib) [ 5499.318496] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5499.518955] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5501.191199] Key type lgssc registered [ 5502.237110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5509.561127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5509.584415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.023326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5514.601810] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5520.358130] LustreError: 176104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5520.369566] LustreError: 176104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5521.817380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.121596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.616199] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5531.644947] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5535.545684] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5540.218588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5540.500168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.461839] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5551.285112] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5555.886044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5561.317785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.338710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5561.345427] Lustre: Skipped 1 previous similar message [ 5566.433307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5566.447871] Lustre: Skipped 1 previous similar message [ 5567.460216] Lustre: server umount lustre-OST0000 complete [ 5571.552215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.554761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.561328] Lustre: Skipped 2 previous similar messages [ 5571.576682] Lustre: Skipped 1 previous similar message [ 5576.674507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5576.689440] Lustre: Skipped 1 previous similar message [ 5577.416218] Lustre: server umount lustre-MDT0000 complete [ 5581.091865] LustreError: 176083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865243 with bad export cookie 10586586568774160704 [ 5581.095165] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5581.103985] LustreError: 176083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5581.379505] Lustre: server umount lustre-MDT0001 complete [ 5591.404982] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5594.088552] Key type lgssc unregistered [ 5594.454540] LNet: 179639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5594.460570] LNetError: 179639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5594.510742] LNet: Removed LNI 192.168.204.112@tcp [ 5595.421379] Key type .llcrypt unregistered [ 5595.423487] Key type ._llcrypt unregistered [ 5620.405848] Key type ._llcrypt registered [ 5620.409176] Key type .llcrypt registered [ 5620.526093] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5621.543529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5621.680924] alg: No test for adler32 (adler32-zlib) [ 5622.815968] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5623.093971] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5624.783335] Key type lgssc registered [ 5625.868139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5637.492703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5637.528987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.943266] Lustre: *** cfs_fail_loc=135, val=0*** [ 5639.048473] LustreError: 181162:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5639.053157] LustreError: 181162:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.112@tcp: cfg command failed: rc = -2 [ 5639.062310] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5639.062310] [ 5639.067401] LustreError: MGC192.168.204.112@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5639.086774] LustreError: 181123:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5639.103778] LustreError: 181123:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5639.114961] LustreError: 181123:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5639.120629] LustreError: 181123:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5639.225202] Lustre: server umount lustre-MDT0000 complete [ 5639.233443] LustreError: 181123:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5646.036952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.529896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.724313] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5657.058049] LustreError: 181535:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.073185] LustreError: 181535:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5657.425495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.696858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5661.429759] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5667.183355] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5673.049668] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.027527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.028980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5677.057469] Lustre: Skipped 1 previous similar message [ 5682.144987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.162943] Lustre: Skipped 2 previous similar messages [ 5683.170206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.182147] Lustre: Skipped 1 previous similar message [ 5688.290692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5688.304543] Lustre: Skipped 1 previous similar message [ 5689.311151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5689.458079] Lustre: server umount lustre-MDT0000 complete [ 5692.557701] LustreError: 182268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865355 with bad export cookie 17719455649163741155 [ 5692.559855] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5692.570874] LustreError: 182268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5692.881333] Lustre: server umount lustre-MDT0001 complete [ 5702.652326] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5705.152870] Key type lgssc unregistered [ 5705.378161] LNet: 183861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5705.383640] LNetError: 183861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5705.394863] LNet: Removed LNI 192.168.204.112@tcp [ 5706.106948] Key type .llcrypt unregistered [ 5706.111326] Key type ._llcrypt unregistered [ 5727.447489] Key type ._llcrypt registered [ 5727.449732] Key type .llcrypt registered [ 5727.602426] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5727.864290] Key type .llcrypt unregistered [ 5727.868052] Key type ._llcrypt unregistered [ 5745.056914] Key type ._llcrypt registered [ 5745.059990] Key type .llcrypt registered [ 5745.228393] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5746.489981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5746.538786] alg: No test for adler32 (adler32-zlib) [ 5747.703832] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5748.009708] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5749.703145] Key type lgssc registered [ 5751.011321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.731653] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5758.751617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.303661] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5779.423329] LustreError: 185998:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.112@tcp: failed processing log, type 4: rc = -110 [ 5806.177783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.092213] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5817.468398] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5823.441879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.933113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.997852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5827.934883] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5829.093948] LustreError: 186024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.124795] LustreError: 186024:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5834.212316] LustreError: 186023:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.945915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.295274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.454590] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5846.743590] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5852.267793] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5855.037832] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5866.981041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.990985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.999776] Lustre: Skipped 1 previous similar message [ 5867.014479] Lustre: Skipped 1 previous similar message [ 5872.103289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5872.114182] Lustre: Skipped 1 previous similar message [ 5872.407725] Lustre: server umount lustre-OST0000 complete [ 5876.192333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5876.201566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5876.226912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5877.218520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.127748] Lustre: server umount lustre-MDT0000 complete [ 5882.346411] LustreError: 187033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.377738] LustreError: 187033:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5885.967337] LustreError: 186005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865548 with bad export cookie 13704800914727415764 [ 5885.968675] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5885.984195] LustreError: 186005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5886.333931] Lustre: server umount lustre-MDT0001 complete [ 5897.512458] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 5900.950348] Key type lgssc unregistered [ 5901.283780] LNet: 189965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5901.302916] LNetError: 189965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5901.337443] LNet: Removed LNI 192.168.204.112@tcp [ 5902.518516] Key type .llcrypt unregistered [ 5902.536087] Key type ._llcrypt unregistered [ 5927.955295] Key type ._llcrypt registered [ 5927.957635] Key type .llcrypt registered [ 5928.068531] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 5929.291824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.330279] alg: No test for adler32 (adler32-zlib) [ 5930.505979] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 5930.992917] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 5932.793306] Key type lgssc registered [ 5934.270300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.376118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.395254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.010591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.434471] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5954.531732] LustreError: 191379:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.569896] LustreError: 191379:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5958.845083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.711697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5965.865880] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5973.346528] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5980.586216] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.101967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.409846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.426774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5993.008651] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 5999.821840] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6005.308321] Lustre: Failing over lustre-MDT0000 [ 6005.561484] Lustre: server umount lustre-MDT0000 complete [ 6005.738314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6005.749655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.771905] LustreError: 192826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.668130] LustreError: 191373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.825109] LustreError: 191379:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6009.831513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.852481] Lustre: Skipped 1 previous similar message [ 6014.952233] LustreError: 191378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.969453] LustreError: 191378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6022.991557] LustreError: 191374:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.016406] LustreError: 191374:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6024.077675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.210860] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6024.449743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6024.508102] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6027.974913] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6028.107949] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6029.803942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6029.850233] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6029.894441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6031.376069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6034.547233] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6045.159694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6045.163903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6045.170878] Lustre: Skipped 1 previous similar message [ 6050.280995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.287308] Lustre: Skipped 2 previous similar messages [ 6050.692479] Lustre: server umount lustre-OST0000 complete [ 6055.393966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6055.395581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.410202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.423509] Lustre: Skipped 1 previous similar message [ 6055.435398] Lustre: Skipped 1 previous similar message [ 6060.343639] Lustre: server umount lustre-MDT0000 complete [ 6060.525157] LustreError: 191375:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6060.544232] LustreError: 191375:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6064.153254] LustreError: 195106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865726 with bad export cookie 14513025295017656474 [ 6064.163309] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6064.166828] LustreError: 195106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6064.430477] Lustre: server umount lustre-MDT0001 complete [ 6075.222943] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6077.667260] Key type lgssc unregistered [ 6077.943961] LNet: 196172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6077.953730] LNetError: 196172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6077.969372] LNet: Removed LNI 192.168.204.112@tcp [ 6078.833282] Key type .llcrypt unregistered [ 6078.835445] Key type ._llcrypt unregistered [ 6098.331357] Key type ._llcrypt registered [ 6098.333807] Key type .llcrypt registered [ 6098.414166] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6099.295171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.324482] alg: No test for adler32 (adler32-zlib) [ 6100.456297] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6100.754267] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6102.463204] Key type lgssc registered [ 6103.450961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6110.672503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6110.688368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.011910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6116.605638] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6121.443906] LustreError: 197565:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.453959] LustreError: 197565:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6124.328169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.714213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.859951] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6137.285939] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6142.918752] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6148.976222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6149.341331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6155.436693] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6156.539495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6162.711184] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6166.480835] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6169.880380] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6187.877993] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6202.850307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.873262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6202.880767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.889126] Lustre: Skipped 1 previous similar message [ 6207.982042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6207.993981] Lustre: Skipped 1 previous similar message [ 6208.982382] Lustre: server umount lustre-OST0000 complete [ 6213.100489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.110449] Lustre: Skipped 1 previous similar message [ 6213.115916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6213.126609] Lustre: Skipped 1 previous similar message [ 6218.215502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6218.228848] Lustre: Skipped 1 previous similar message [ 6219.449290] Lustre: server umount lustre-MDT0000 complete [ 6222.768500] LustreError: 197546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787865885 with bad export cookie 8643432890893081294 [ 6222.777183] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.785730] LustreError: 197546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6223.017686] Lustre: server umount lustre-MDT0001 complete [ 6233.973309] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6236.660357] Key type lgssc unregistered [ 6236.960789] LNet: 201458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6236.968299] LNetError: 201458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6236.984243] LNet: Removed LNI 192.168.204.112@tcp [ 6237.919830] Key type .llcrypt unregistered [ 6237.922144] Key type ._llcrypt unregistered [ 6255.381759] Key type ._llcrypt registered [ 6255.383585] Key type .llcrypt registered [ 6255.468288] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6256.216154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6256.276482] alg: No test for adler32 (adler32-zlib) [ 6257.345323] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6257.594788] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6259.303261] Key type lgssc registered [ 6260.392259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6267.885623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6267.940696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.332960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.481302] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6279.654055] LustreError: 202726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.689834] LustreError: 202726:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6280.780033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.109580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6285.101597] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6291.933470] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6298.400861] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6305.258724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6305.634606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.035261] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6314.874977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6319.675405] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6324.860818] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6338.466764] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6348.037884] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6356.148771] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6368.237426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6368.245350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6368.257294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.304811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.310416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.341993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.506249] Lustre: server umount lustre-OST0000 complete [ 6381.536444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.548597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6381.553696] Lustre: Skipped 2 previous similar messages [ 6385.089963] Lustre: server umount lustre-MDT0000 complete [ 6386.674671] LustreError: 203453:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6389.428830] LustreError: 205403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866052 with bad export cookie 10249670836868740273 [ 6389.434188] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6389.437061] LustreError: 205403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6389.736195] Lustre: server umount lustre-MDT0001 complete [ 6401.178327] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6403.954598] Key type lgssc unregistered [ 6404.342330] LNet: 207751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6404.355430] LNetError: 207751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6404.379685] LNet: Removed LNI 192.168.204.112@tcp [ 6405.137232] Key type .llcrypt unregistered [ 6405.150828] Key type ._llcrypt unregistered [ 6427.989995] Key type ._llcrypt registered [ 6427.992343] Key type .llcrypt registered [ 6428.120395] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6429.169671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6429.423582] alg: No test for adler32 (adler32-zlib) [ 6430.504007] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6430.844866] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6432.543973] Key type lgssc registered [ 6434.102225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6443.099240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6443.120599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.575874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.207859] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6454.753709] LustreError: 209166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6454.775610] LustreError: 209166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6457.691132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.334690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6463.376700] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6471.652361] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6477.945387] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6484.915432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.287672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6489.404460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6491.897242] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6500.852583] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6519.264312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6519.268973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6519.297562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6519.305695] Lustre: Skipped 1 previous similar message [ 6524.394206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6524.407599] Lustre: Skipped 2 previous similar messages [ 6525.049777] Lustre: server umount lustre-OST0000 complete [ 6529.516289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.531050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6529.540706] Lustre: Skipped 1 previous similar message [ 6529.568918] Lustre: Skipped 1 previous similar message [ 6534.631889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6534.639738] Lustre: Skipped 1 previous similar message [ 6539.749416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6539.755461] Lustre: Skipped 1 previous similar message [ 6543.327337] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.584152] Lustre: server umount lustre-MDT0000 complete [ 6544.867105] LustreError: 209166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.904445] LustreError: 209166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6548.783312] LustreError: 209901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866211 with bad export cookie 9324794531260876176 [ 6548.784824] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6548.817072] LustreError: 209901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6549.127354] Lustre: server umount lustre-MDT0001 complete [ 6561.701848] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6564.880634] Key type lgssc unregistered [ 6565.210130] LNet: 212856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6565.222585] LNetError: 212856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6565.263295] LNet: Removed LNI 192.168.204.112@tcp [ 6566.345996] Key type .llcrypt unregistered [ 6566.350640] Key type ._llcrypt unregistered [ 6586.861593] Key type ._llcrypt registered [ 6586.867114] Key type .llcrypt registered [ 6586.976913] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6588.189445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6588.241515] alg: No test for adler32 (adler32-zlib) [ 6589.321777] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6589.595601] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6591.303335] Key type lgssc registered [ 6592.525959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6599.996374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6600.024536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.447625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.038410] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6611.938515] LustreError: 214122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.972256] LustreError: 214122:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6616.384519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.918343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6621.925576] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6631.001535] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6637.574659] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.335112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.783288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.921380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6654.678814] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6662.993205] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6677.478865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6677.480910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.486579] LustreError: Skipped 1 previous similar message [ 6677.493510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6677.497645] Lustre: Skipped 1 previous similar message [ 6682.593773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.602971] Lustre: Skipped 1 previous similar message [ 6683.375629] Lustre: server umount lustre-OST0000 complete [ 6687.720422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.738842] Lustre: Skipped 1 previous similar message [ 6687.741681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6687.750351] Lustre: Skipped 1 previous similar message [ 6692.834332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6692.853730] Lustre: Skipped 1 previous similar message [ 6693.026374] Lustre: server umount lustre-MDT0000 complete [ 6697.106293] LustreError: 214103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866359 with bad export cookie 10714268580100837800 [ 6697.110907] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6697.118917] LustreError: 214103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6697.548868] Lustre: server umount lustre-MDT0001 complete [ 6709.279673] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6712.465887] Key type lgssc unregistered [ 6712.833799] LNet: 217811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6712.858177] LNetError: 217811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6713.893954] LNet: Removed LNI 192.168.204.112@tcp [ 6714.879431] Key type .llcrypt unregistered [ 6714.882773] Key type ._llcrypt unregistered [ 6735.704696] Key type ._llcrypt registered [ 6735.707630] Key type .llcrypt registered [ 6735.807832] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6736.953386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6737.005307] alg: No test for adler32 (adler32-zlib) [ 6738.008518] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6738.279601] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6739.992453] Key type lgssc registered [ 6741.229114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6750.777591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6750.798921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6752.384347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6758.107139] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6762.979243] LustreError: 219077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6763.003911] LustreError: 219077:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6768.098325] LustreError: 219078:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.801343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.217829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6774.796949] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6783.436441] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6790.798321] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6799.135861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6799.688544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.982683] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6816.049454] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6823.218446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6830.798642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6830.862271] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6831.148989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6837.758913] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 6845.798160] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6847.098412] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6859.017746] Lustre: Permanently deactivating lustre-OST0001 [ 6859.035064] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6867.116970] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6877.289883] Lustre: Permanently reactivating lustre-OST0001 [ 6877.306648] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6877.314570] Lustre: Skipped 2 previous similar messages [ 6886.953418] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.966983] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6886.974896] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6886.994182] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6887.058384] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6887.077257] LustreError: lustre-OST0001-osc-MDT0000: operation ost_get_info to node 0@lo failed: rc = -107 [ 6887.078398] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6887.085883] LustreError: Skipped 1 previous similar message [ 6889.253767] Lustre: server umount lustre-OST0001 complete [ 6892.525247] LustreError: 220899:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.552312] LustreError: 220899:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6895.073534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6895.081406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.098215] Lustre: Skipped 3 previous similar messages [ 6895.106210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.109693] Lustre: Skipped 1 previous similar message [ 6897.121274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.130112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.145952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6897.640616] LustreError: 220901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.655549] LustreError: 220901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6898.780818] Lustre: server umount lustre-OST0000 complete [ 6902.758065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6902.767134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.769677] Lustre: Skipped 1 previous similar message [ 6902.786506] Lustre: Skipped 2 previous similar messages [ 6907.881561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.885144] Lustre: Skipped 1 previous similar message [ 6908.164832] Lustre: server umount lustre-MDT0000 complete [ 6911.639993] LustreError: 219058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866574 with bad export cookie 12497520348019160690 [ 6911.645399] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6911.654959] LustreError: 219058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6912.120586] Lustre: server umount lustre-MDT0001 complete [ 6923.423534] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 6927.009295] Key type lgssc unregistered [ 6927.422818] LNet: 224175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6927.452259] LNetError: 224175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6927.482787] LNet: Removed LNI 192.168.204.112@tcp [ 6928.668812] Key type .llcrypt unregistered [ 6928.677234] Key type ._llcrypt unregistered [ 6976.876860] Key type ._llcrypt registered [ 6976.881744] Key type .llcrypt registered [ 6977.016178] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 6978.602175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6978.636180] alg: No test for adler32 (adler32-zlib) [ 6979.855494] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 6980.151597] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 6981.791184] Key type lgssc registered [ 6982.996648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6990.698961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6990.722460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.953652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6991.982258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6992.186240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.394102] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7004.994272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.185594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7005.228479] Lustre: 226382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7005.366180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7005.547815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7009.672163] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7017.038657] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7022.439871] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7028.222409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7028.406758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7028.589280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.244713] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7037.014088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7042.086772] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7046.155355] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7052.552381] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7061.500620] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7068.164842] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7083.746985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7083.758071] Lustre: Skipped 1 previous similar message [ 7105.920325] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7105.924881] Lustre: Skipped 2 previous similar messages [ 7141.549551] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7141.560776] Lustre: Skipped 4 previous similar messages [ 7210.085685] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7210.091707] Lustre: Skipped 7 previous similar messages [ 7219.174059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.175368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.192439] Lustre: Skipped 1 previous similar message [ 7219.201366] Lustre: Skipped 1 previous similar message [ 7223.821850] Lustre: server umount lustre-OST0000 complete [ 7229.410529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.447139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7229.452264] Lustre: Skipped 1 previous similar message [ 7234.530856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7234.541600] Lustre: Skipped 1 previous similar message [ 7239.658751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7239.690709] Lustre: Skipped 1 previous similar message [ 7243.231405] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7243.597219] Lustre: server umount lustre-MDT0000 complete [ 7247.334835] LustreError: 229176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787866910 with bad export cookie 12106345785752971943 [ 7247.351962] LustreError: 229176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7247.365660] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.681185] Lustre: server umount lustre-MDT0001 complete [ 7259.802557] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 7263.178459] Key type lgssc unregistered [ 7263.527354] LNet: 231185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7263.534471] LNetError: 231185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7263.551578] LNet: Removed LNI 192.168.204.112@tcp [ 7264.564132] Key type .llcrypt unregistered [ 7264.566991] Key type ._llcrypt unregistered [ 7292.773422] Key type ._llcrypt registered [ 7292.778609] Key type .llcrypt registered [ 7293.006945] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7293.858344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7293.897526] alg: No test for adler32 (adler32-zlib) [ 7294.964517] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 7295.232324] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 7296.984412] Key type lgssc registered [ 7298.085699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7304.335454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7304.362523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.828731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7309.606573] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7315.937065] LustreError: 232601:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.983961] LustreError: 232601:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7316.684593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.120661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7321.524791] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7329.669863] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7335.017879] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7341.725756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7342.261322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7347.562873] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_default_debug -1 all [ 7350.770267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7354.016084] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7371.232820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.238884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.251450] Lustre: Skipped 1 previous similar message [ 7371.252563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7371.269913] Lustre: Skipped 1 previous similar message [ 7371.911358] Lustre: server umount lustre-OST0000 complete [ 7376.361445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.370850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7376.379602] Lustre: Skipped 1 previous similar message [ 7376.392813] Lustre: Skipped 1 previous similar message [ 7381.478682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7381.482558] Lustre: Skipped 1 previous similar message [ 7381.564782] Lustre: server umount lustre-MDT0000 complete [ 7385.050283] LustreError: 232582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787867047 with bad export cookie 12030550590853712982 [ 7385.062974] LustreError: MGC192.168.204.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.069994] LustreError: 232582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7385.470196] Lustre: server umount lustre-MDT0001 complete [ 7396.093949] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 7398.904663] Key type lgssc unregistered [ 7399.215740] LNet: 236294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7399.227467] LNetError: 236294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7399.256358] LNet: Removed LNI 192.168.204.112@tcp [ 7400.489242] Key type .llcrypt unregistered [ 7400.491679] Key type ._llcrypt unregistered [ 7426.015230] Key type ._llcrypt registered [ 7426.019638] Key type .llcrypt registered [ 7426.121465] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7427.353824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7427.442072] alg: No test for adler32 (adler32-zlib) [ 7428.466926] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 7428.720704] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 7430.375230] Key type lgssc registered [ 7431.447837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7699.536780] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 7702.409137] Key type lgssc unregistered [ 7702.703902] LNet: 238321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7702.708474] LNetError: 238321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7702.726225] LNet: Removed LNI 192.168.204.112@tcp [ 7703.600401] Key type .llcrypt unregistered [ 7703.608616] Key type ._llcrypt unregistered [ 7728.013133] Key type ._llcrypt registered [ 7728.014902] Key type .llcrypt registered [ 7728.129125] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing set_hostid [ 7744.826931] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing load_modules_local [ 7746.011550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7746.130810] alg: No test for adler32 (adler32-zlib) [ 7747.180227] Lustre: Lustre: Build Version: 2.17.57_124_g5f91490 [ 7747.441194] LNet: Added LNI 192.168.204.112@tcp [8/256/0/180] [ 7749.151733] Key type lgssc registered [ 7750.486164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7785.668357] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7499 sec ========== 17:50:46 (1787867446) [ 7787.330902] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:50:48 (1787867448) === [ 7790.633447] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:50:51 (1787867451) === [ 7831.263235] Lustre: DEBUG MARKER: oleg412-server.virtnet: executing unload_modules_local [ 7833.784428] Key type lgssc unregistered [ 7834.113489] LNet: 243561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7834.123927] LNetError: 243561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7834.145060] LNet: Removed LNI 192.168.204.112@tcp [ 7835.030499] Key type .llcrypt unregistered [ 7835.033675] Key type ._llcrypt unregistered