[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 636332637 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002333] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004027] kvm-guest: setup PV IPIs [ 0.007366] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008042] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.011045] LSM: Security Framework initializing [ 0.012070] Yama: becoming mindful. [ 0.013062] SELinux: Initializing. [ 0.015007] *** VALIDATE selinux *** [ 0.023427] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028263] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031087] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032137] *** VALIDATE tmpfs *** [ 0.033483] *** VALIDATE proc *** [ 0.035210] *** VALIDATE cgroup *** [ 0.036014] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044296] debug: unmapping init [mem 0xffffffff91a59000-0xffffffff91a60fff] [ 0.047185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048951] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050030] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052013] ... generic registers: 4 [ 0.053011] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057368] rcu: Hierarchical SRCU implementation. [ 0.059551] smp: Bringing up secondary CPUs ... [ 0.060600] x86: Booting SMP configuration: [ 0.061032] .... node #0, CPUs: #1 #2 #3 [ 0.064777] smp: Brought up 1 node, 4 CPUs [ 0.066031] smpboot: Max logical packages: 1 [ 0.067030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236287] node 0 deferred pages initialised in 165ms [ 0.239144] devtmpfs: initialized [ 0.240277] x86/mm: Memory block size: 128MB [ 0.243946] gcov: version magic: 0x41383552 [ 0.247257] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.253283] pinctrl core: initialized pinctrl subsystem [ 0.256209] [ 0.256872] ************************************************************* [ 0.259016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262017] ** ** [ 0.264012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267019] ** ** [ 0.269013] ** This means that this kernel is built to expose internal ** [ 0.272014] ** IOMMU data structures, which may compromise security on ** [ 0.275016] ** your system. ** [ 0.277012] ** ** [ 0.279012] ** If you see this message and you are not debugging the ** [ 0.283024] ** kernel, report this immediately to your vendor! ** [ 0.286021] ** ** [ 0.289018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292014] ************************************************************* [ 0.295954] NET: Registered protocol family 16 [ 0.298672] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306109] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.311040] cpuidle: using governor menu [ 0.312820] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313549] PCI: Using configuration type 1 for base access [ 0.314137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.325023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.331118] cryptd: max_cpu_qlen set to 1000 [ 0.336331] ACPI: Added _OSI(Module Device) [ 0.338018] ACPI: Added _OSI(Processor Device) [ 0.340016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.347130] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355607] ACPI: Interpreter enabled [ 0.357081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.358019] ACPI: Using IOAPIC for interrupt routing [ 0.360155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.364440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.374754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.377060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.380029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.384119] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.390548] acpiphp: Slot [2] registered [ 0.392215] acpiphp: Slot [5] registered [ 0.394208] acpiphp: Slot [6] registered [ 0.396160] acpiphp: Slot [7] registered [ 0.398590] acpiphp: Slot [8] registered [ 0.400214] acpiphp: Slot [9] registered [ 0.403186] acpiphp: Slot [10] registered [ 0.405194] acpiphp: Slot [3] registered [ 0.407131] acpiphp: Slot [4] registered [ 0.409119] acpiphp: Slot [11] registered [ 0.410122] acpiphp: Slot [12] registered [ 0.412109] acpiphp: Slot [13] registered [ 0.414141] acpiphp: Slot [14] registered [ 0.415156] acpiphp: Slot [15] registered [ 0.417111] acpiphp: Slot [16] registered [ 0.419095] acpiphp: Slot [17] registered [ 0.420161] acpiphp: Slot [18] registered [ 0.422175] acpiphp: Slot [19] registered [ 0.424151] acpiphp: Slot [20] registered [ 0.426159] acpiphp: Slot [21] registered [ 0.428107] acpiphp: Slot [22] registered [ 0.430139] acpiphp: Slot [23] registered [ 0.432164] acpiphp: Slot [24] registered [ 0.434175] acpiphp: Slot [25] registered [ 0.436181] acpiphp: Slot [26] registered [ 0.438194] acpiphp: Slot [27] registered [ 0.440227] acpiphp: Slot [28] registered [ 0.442168] acpiphp: Slot [29] registered [ 0.444137] acpiphp: Slot [30] registered [ 0.445131] acpiphp: Slot [31] registered [ 0.447104] PCI host bridge to bus 0000:00 [ 0.449026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.452043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.454027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.457037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.461066] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.464042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.466223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.470174] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.474471] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.484022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.489012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.492034] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.495026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.499029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.503081] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.506875] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.510055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.513872] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.520023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.535021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.541919] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.547743] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.557019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.564021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.586021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.598399] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.604021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.611019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.626022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.636032] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.642017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.648017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.663021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.675436] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.682018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.687017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.704033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.720648] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.735019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.756024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.776077] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.785019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.793020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.817031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.829043] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.832454] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.835499] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.840307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.846151] iommu: Default domain type: Passthrough [ 0.848513] SCSI subsystem initialized [ 0.850145] ACPI: bus type USB registered [ 0.852140] usbcore: registered new interface driver usbfs [ 0.854091] usbcore: registered new interface driver hub [ 0.856090] usbcore: registered new device driver usb [ 0.858214] pps_core: LinuxPPS API ver. 1 registered [ 0.861017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.864071] PTP clock support registered [ 0.867124] EDAC MC: Ver: 3.0.0 [ 0.869118] PCI: Using ACPI for IRQ routing [ 0.872055] NetLabel: Initializing [ 0.874015] NetLabel: domain hash size = 128 [ 0.876013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.878104] NetLabel: unlabeled traffic allowed by default [ 0.880191] vgaarb: loaded [ 0.882314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.884014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.891217] clocksource: Switched to clocksource kvm-clock [ 0.997413] VFS: Disk quotas dquot_6.6.0 [ 0.999480] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.002726] *** VALIDATE ramfs *** [ 1.003839] *** VALIDATE hugetlbfs *** [ 1.005761] pnp: PnP ACPI init [ 1.008456] pnp: PnP ACPI: found 6 devices [ 1.033303] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.037260] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.039469] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.041976] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.044890] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.047856] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.051285] NET: Registered protocol family 2 [ 1.054113] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.059667] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.063806] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.069848] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.073928] TCP: Hash tables configured (established 65536 bind 65536) [ 1.077393] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.081270] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.084371] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.087787] NET: Registered protocol family 1 [ 1.090996] RPC: Registered named UNIX socket transport module. [ 1.093687] RPC: Registered udp transport module. [ 1.094916] RPC: Registered tcp transport module. [ 1.096773] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.098744] NET: Registered protocol family 44 [ 1.099968] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.102402] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.104685] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.108952] PCI: CLS 0 bytes, default 64 [ 1.111377] Unpacking initramfs... [ 2.559559] debug: unmapping init [mem 0xffff8adebcc54000-0xffff8adebffbffff] [ 2.566645] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.569509] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.574304] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.068604] Initialise system trusted keyrings [ 3.070397] Key type blacklist registered [ 3.073164] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.082960] zbud: loaded [ 3.086236] *** VALIDATE nfs *** [ 3.087701] *** VALIDATE nfs4 *** [ 3.089493] pstore: using deflate compression [ 3.093608] Platform Keyring initialized [ 3.202889] NET: Registered protocol family 38 [ 3.204949] Key type asymmetric registered [ 3.206943] Asymmetric key parser 'x509' registered [ 3.209309] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.212900] io scheduler mq-deadline registered [ 3.214909] io scheduler kyber registered [ 3.216582] io scheduler bfq registered [ 3.218707] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.222721] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.226462] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.229430] ACPI: Power Button [PWRF] [ 3.234718] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.241694] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.254201] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.261106] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.274975] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.303463] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.332751] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.337751] Non-volatile memory driver v1.3 [ 3.339602] Linux agpgart interface v0.103 [ 3.371491] virtio_blk virtio1: [vda] 146872 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.375163] vda: detected capacity change from 0 to 75198464 [ 3.400899] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.406635] vdb: detected capacity change from 0 to 1073741824 [ 3.429359] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.431966] vdc: detected capacity change from 0 to 2621440000 [ 3.449406] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.452759] vdd: detected capacity change from 0 to 2621440000 [ 3.471042] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.474459] vde: detected capacity change from 0 to 4294967296 [ 3.489525] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.493071] vdf: detected capacity change from 0 to 4294967296 [ 3.504051] libphy: Fixed MDIO Bus: probed [ 3.509974] usbcore: registered new interface driver usbserial_generic [ 3.514785] usbserial: USB Serial support registered for generic [ 3.517721] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.523306] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.525981] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.528551] mousedev: PS/2 mouse device common for all mice [ 3.531315] rtc_cmos 00:05: RTC can wake from S4 [ 3.535196] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.535551] rtc_cmos 00:05: registered as rtc0 [ 3.541413] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.541472] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.547394] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.547719] intel_pstate: CPU model not supported [ 3.554467] hid: raw HID events driver (C) Jiri Kosina [ 3.557067] usbcore: registered new interface driver usbhid [ 3.559511] usbhid: USB HID core driver [ 3.561451] drop_monitor: Initializing network drop monitor service [ 3.564432] Initializing XFRM netlink socket [ 3.566880] NET: Registered protocol family 10 [ 3.570232] Segment Routing with IPv6 [ 3.571763] NET: Registered protocol family 17 [ 3.573480] mpls_gso: MPLS GSO support [ 3.580310] RAS: Correctable Errors collector initialized. [ 3.582979] AVX version of gcm_enc/dec engaged. [ 3.585383] AES CTR mode by8 optimization enabled [ 3.662899] sched_clock: Marking stable (3662860323, 0)->(4648570310, -985709987) [ 3.668080] registered taskstats version 1 [ 3.671057] Loading compiled-in X.509 certificates [ 3.674529] zswap: loaded using pool lzo/zbud [ 3.702123] Key type big_key registered [ 3.714846] Key type encrypted registered [ 3.717950] ima: No TPM chip found, activating TPM-bypass! [ 3.723154] ima: Allocated hash algorithm: sha1 [ 3.727239] ima: No architecture policies found [ 3.731256] evm: Initialising EVM extended attributes: [ 3.734641] evm: security.selinux [ 3.737575] evm: security.ima [ 3.738877] evm: security.capability [ 3.741558] evm: HMAC attrs: 0x1 [ 3.745934] rtc_cmos 00:05: setting system clock to 2026-08-27 16:12:52 UTC (1787847172) [ 3.754654] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 3.758682] debug: unmapping init [mem 0xffffffff91782000-0xffffffff91a58fff] [ 3.768085] Write protecting the kernel read-only data: 28672k [ 3.771849] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 3.774832] debug: unmapping init [mem 0xffffffff90714000-0xffffffff907fffff] [ 3.810718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.819934] systemd[1]: Detected virtualization kvm. [ 3.821845] systemd[1]: Detected architecture x86-64. [ 3.823765] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.850088] systemd[1]: No hostname configured. [ 3.852077] systemd[1]: Set hostname to . [ 3.854656] random: systemd: uninitialized urandom read (16 bytes read) [ 3.857667] systemd[1]: Initializing machine ID from random generator. [ 3.902826] random: ln: uninitialized urandom read (6 bytes read) [ 4.012579] random: systemd: uninitialized urandom read (16 bytes read) [ 4.015494] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.021795] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.027247] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.716482] device-mapper: uevent: version 1.0.3 [ 4.718549] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Ke[ 4.986331] random: fast init done rnel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.473707] virtio_net virtio0 ens2: renamed from eth0 [ 5.519361] scsi host0: ata_piix [ 5.534216] scsi host1: ata_piix [ 5.535658] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.539525] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.229027] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.249526] random: crng init done [ 10.251261] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.694241] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.898946] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.154486] SELinux: Disabled at runtime. [ 12.221470] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.231679] systemd[1]: Detected virtualization kvm. [ 12.233307] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.756606] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.760334] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.766261] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.770432] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.773177] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.782305] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.790606] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.991546] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.247105] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.508764] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.574204] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.694454] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.721390] EDAC sbridge: Ver: 1.1.2 [ 15.335287] Key type dns_resolver registered [ 15.649472] NFS: Registering the id_resolver key type [ 15.652387] Key type id_resolver registered [ 15.654340] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 41.786057] libcfs: loading out-of-tree module taints kernel. [ 41.809219] Key type ._llcrypt registered [ 41.811156] Key type .llcrypt registered [ 41.861533] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 51.159462] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 51.953623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.963392] alg: No test for adler32 (adler32-zlib) [ 53.006695] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 53.371335] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 55.016277] Key type lgssc registered [ 56.781814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.893032] hrtimer: interrupt took 25036401 ns [ 73.143510] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.372349] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 131.994754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.074145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.234899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.280788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.402578] Lustre: lustre-MDT0000: new disk, initializing [ 133.499485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.524212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.418989] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.955708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.094561] Lustre: 6481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.117743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.124066] Lustre: Skipped 1 previous similar message [ 152.190724] Lustre: lustre-MDT0001: new disk, initializing [ 152.234887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 152.265621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 152.282089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 156.777223] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.880321] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.020568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.227897] Lustre: lustre-OST0000: new disk, initializing [ 172.234595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.247355] Lustre: 8420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.316478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.999667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 173.014593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 173.081330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.158909] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.366660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.551297] Lustre: lustre-OST0001: new disk, initializing [ 194.557617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.573872] Lustre: 9493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.686386] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.845971] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.338480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.352589] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.418755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 215.607805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.457210] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 232.883725] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.906090] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 243.947853] Lustre: DEBUG MARKER: Client: 2.17.57.86 [ 247.490376] Lustre: DEBUG MARKER: MDS: 2.17.57.86 [ 250.005664] Lustre: DEBUG MARKER: OSS: 2.17.57.86 [ 253.018634] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:16:59 EDT 2026 [ 275.731336] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 278.428903] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.905494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 295.916684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.928610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.418690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.442627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.449959] Lustre: Skipped 1 previous similar message [ 301.537872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.549974] Lustre: Skipped 3 previous similar messages [ 301.929076] Lustre: server umount lustre-MDT0000 complete [ 305.618989] LustreError: 8419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847474 with bad export cookie 15258072082831173227 [ 305.619122] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.634137] LustreError: 8419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 306.657762] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.658581] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.681064] Lustre: Skipped 3 previous similar messages [ 306.701382] Lustre: Skipped 1 previous similar message [ 310.697608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.704242] Lustre: Skipped 1 previous similar message [ 312.095884] Lustre: server umount lustre-MDT0001 complete [ 316.577470] Lustre: server umount lustre-OST0000 complete [ 320.547574] Lustre: server umount lustre-OST0001 complete [ 327.695270] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 338.511637] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 379.696439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.934566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 379.970268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.050082] Lustre: lustre-MDT0000: new disk, initializing [ 380.118286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.143295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.654183] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 395.542955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.698063] Lustre: 15048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.791494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.797135] Lustre: Skipped 1 previous similar message [ 395.868500] Lustre: lustre-MDT0001: new disk, initializing [ 395.988764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.054808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.071871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.907165] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 409.605945] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 416.423265] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.083336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.467552] Lustre: lustre-OST0000: new disk, initializing [ 424.470992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.477702] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.626726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.178864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.196437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.321638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 431.410800] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 442.182658] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 448.341329] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 448.559495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.031867] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.277771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.186375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.205480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.209272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.231799] Lustre: Skipped 1 previous similar message [ 462.561895] Lustre: server umount lustre-OST0000 complete [ 467.427414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.441691] Lustre: Skipped 1 previous similar message [ 473.429261] Lustre: server umount lustre-MDT0000 complete [ 477.558057] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847646 with bad export cookie 15258072082831174760 [ 477.570942] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.574317] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.896630] Lustre: server umount lustre-MDT0001 complete [ 481.042426] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:20:48 (1787847648) [ 486.950475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.458832] LustreError: 19282:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.477303] LustreError: 19282:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 487.546926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.003318] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 493.029772] LustreError: 19283:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.161528] LustreError: 19282:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.395237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 507.688527] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 515.438700] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.146311] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.992083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 537.052825] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 544.215569] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.004628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 556.023994] LustreError: Skipped 1 previous similar message [ 556.028643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.048015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.066692] Lustre: Skipped 7 previous similar messages [ 560.976891] Lustre: server umount lustre-OST0000 complete [ 570.846294] Lustre: server umount lustre-MDT0000 complete [ 572.393776] LustreError: 20238:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.416090] LustreError: 20238:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 575.137367] LustreError: 21125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847743 with bad export cookie 15258072082831175642 [ 575.159218] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.161891] LustreError: 21125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 575.495868] Lustre: server umount lustre-MDT0001 complete [ 588.627074] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 592.048118] Key type lgssc unregistered [ 592.369597] LNet: 22818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.377594] LNetError: 22818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.409886] LNet: Removed LNI 192.168.203.149@tcp [ 593.214451] Key type .llcrypt unregistered [ 593.220092] Key type ._llcrypt unregistered [ 623.530075] Key type ._llcrypt registered [ 623.531806] Key type .llcrypt registered [ 623.696872] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 625.373954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 625.607542] alg: No test for adler32 (adler32-zlib) [ 626.724240] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 626.976525] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 628.632144] Key type lgssc registered [ 630.031867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 638.580838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 638.601155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.037645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.506781] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 650.222301] LustreError: 24233:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 650.262688] LustreError: 24233:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 655.335775] LustreError: 24234:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.437762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.804308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 661.236258] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 668.266763] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 674.101486] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 680.429840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.828615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.941683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 687.190275] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 694.413602] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 710.625312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 710.633226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.640381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.652712] Lustre: Skipped 1 previous similar message [ 712.182263] Lustre: server umount lustre-OST0000 complete [ 718.305410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.313264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.328249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.337427] Lustre: Skipped 1 previous similar message [ 720.865825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 720.868635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.229248] Lustre: server umount lustre-MDT0000 complete [ 725.891215] LustreError: 24214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847894 with bad export cookie 12853895083493326623 [ 725.895685] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.904535] LustreError: 24214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 726.289570] Lustre: server umount lustre-MDT0001 complete [ 736.346959] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 739.336191] Key type lgssc unregistered [ 739.722939] LNet: 28213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.731241] LNetError: 28213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.755272] LNet: Removed LNI 192.168.203.149@tcp [ 740.849447] Key type .llcrypt unregistered [ 740.852659] Key type ._llcrypt unregistered [ 763.236038] Key type ._llcrypt registered [ 763.239222] Key type .llcrypt registered [ 763.386379] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 764.321423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.458295] alg: No test for adler32 (adler32-zlib) [ 765.439616] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 765.652692] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 767.272116] Key type lgssc registered [ 768.582701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 775.664519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 775.690712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.988258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.771574] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 786.406610] LustreError: 29604:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.417790] LustreError: 29604:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 789.149894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.451922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 793.126198] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 799.867298] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 806.556553] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 820.519031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 820.770142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.814384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 827.929254] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 835.546349] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 845.285312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.289688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 845.303447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 845.319380] Lustre: Skipped 1 previous similar message [ 848.708814] Lustre: server umount lustre-OST0000 complete [ 855.524612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.531674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.539751] Lustre: Skipped 2 previous similar messages [ 858.629337] Lustre: server umount lustre-MDT0000 complete [ 860.641704] LustreError: 30332:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.664317] LustreError: 30332:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 861.923660] LustreError: 29586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848030 with bad export cookie 5684220289147108164 [ 861.929032] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.936872] LustreError: 29586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 862.307449] Lustre: server umount lustre-MDT0001 complete [ 874.160473] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 876.759839] Key type lgssc unregistered [ 877.047815] LNet: 33486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.063054] LNetError: 33486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.093782] LNet: Removed LNI 192.168.203.149@tcp [ 877.818853] Key type .llcrypt unregistered [ 877.821240] Key type ._llcrypt unregistered [ 903.410763] Key type ._llcrypt registered [ 903.413607] Key type .llcrypt registered [ 903.535144] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 904.441272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.574457] alg: No test for adler32 (adler32-zlib) [ 905.699869] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 905.989682] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 907.680457] Key type lgssc registered [ 908.820360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.983566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.998086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.505349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.231717] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 927.716149] LustreError: 34902:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.733817] LustreError: 34902:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 931.465055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.788756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.942184] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 943.921228] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 950.003468] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 955.520837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.878562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.545333] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 963.057661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 968.370612] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 977.380212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 977.395147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.419224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.403341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 978.404116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.521620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 983.535139] Lustre: Skipped 2 previous similar messages [ 988.644561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.648021] Lustre: Skipped 1 previous similar message [ 991.201550] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 991.370721] Lustre: server umount lustre-OST0000 complete [ 998.383129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.393233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.410652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.121577] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1009.124472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.153258] Lustre: Skipped 4 previous similar messages [ 1009.468623] Lustre: server umount lustre-MDT0000 complete [ 1013.031547] LustreError: 34882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848181 with bad export cookie 5651651769958186119 [ 1013.041831] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.049349] LustreError: 34882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1013.377594] Lustre: server umount lustre-MDT0001 complete [ 1025.992809] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1028.915769] Key type lgssc unregistered [ 1029.198256] LNet: 38444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.214779] LNetError: 38444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.230838] LNet: Removed LNI 192.168.203.149@tcp [ 1030.115503] Key type .llcrypt unregistered [ 1030.118116] Key type ._llcrypt unregistered [ 1052.446982] Key type ._llcrypt registered [ 1052.450022] Key type .llcrypt registered [ 1052.526208] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1053.481679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.515393] alg: No test for adler32 (adler32-zlib) [ 1054.549626] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1054.778978] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1056.408174] Key type lgssc registered [ 1057.397517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1064.089959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1064.125292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.598761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.484464] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1076.193366] LustreError: 39861:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1076.224776] LustreError: 39861:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1077.233318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.551860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.699360] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1088.813412] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.324936] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.224148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.797420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.986904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1111.817615] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1119.729080] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.492434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1131.501070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.522113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1132.052677] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 1133.539683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.548497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.559615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1133.572029] Lustre: Skipped 1 previous similar message [ 1137.128064] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 1142.249235] Lustre: lustre-OST0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 1142.261838] Lustre: Skipped 2 previous similar messages [ 1143.264558] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1143.435952] Lustre: server umount lustre-OST0000 complete [ 1162.742810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.745990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1162.769854] Lustre: Skipped 1 previous similar message [ 1162.784049] Lustre: Skipped 1 previous similar message [ 1177.057869] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1177.328104] Lustre: server umount lustre-MDT0000 complete [ 1179.108911] LustreError: 39856:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.132554] LustreError: 39856:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1180.907369] LustreError: 39844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848349 with bad export cookie 9193862958384905240 [ 1180.916355] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.923738] LustreError: 39844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1187.399521] Lustre: server umount lustre-MDT0001 complete [ 1198.137333] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1200.953657] Key type lgssc unregistered [ 1201.447671] LNet: 43573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.461975] LNetError: 43573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.473700] LNet: Removed LNI 192.168.203.149@tcp [ 1202.356628] Key type .llcrypt unregistered [ 1202.360417] Key type ._llcrypt unregistered [ 1226.715821] Key type ._llcrypt registered [ 1226.718168] Key type .llcrypt registered [ 1226.830603] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1228.379975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.393401] alg: No test for adler32 (adler32-zlib) [ 1229.580591] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1229.970595] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1231.712262] Key type lgssc registered [ 1233.024977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.628527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.659225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.209727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.656930] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1253.355573] LustreError: 44985:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.385879] LustreError: 44985:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1257.978082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.208134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.979317] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1268.664255] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1274.334802] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1280.807249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.067568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.893380] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1289.268803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1295.669518] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1302.474993] Lustre: server umount lustre-MDT0000 complete [ 1303.550451] LustreError: 44981:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.553622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.563695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1304.593750] LustreError: 46226:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.632305] LustreError: 44967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848474 with bad export cookie 16667582397760781688 [ 1305.637603] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1305.648548] LustreError: 44967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1308.640583] LustreError: 44982:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.664788] LustreError: 44982:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1308.687170] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 1309.674992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.686921] Lustre: Skipped 2 previous similar messages [ 1309.697665] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1312.015499] Lustre: server umount lustre-MDT0001 complete [ 1333.245771] Lustre: server umount lustre-OST0000 complete [ 1350.263615] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1352.836800] Key type lgssc unregistered [ 1353.128709] LNet: 48850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.136764] LNetError: 48850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1353.156343] LNet: Removed LNI 192.168.203.149@tcp [ 1354.106233] Key type .llcrypt unregistered [ 1354.108954] Key type ._llcrypt unregistered [ 1375.618345] Key type ._llcrypt registered [ 1375.621021] Key type .llcrypt registered [ 1375.711722] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1376.783809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.872511] alg: No test for adler32 (adler32-zlib) [ 1377.886088] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1378.113279] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1379.809422] Key type lgssc registered [ 1380.791478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.416457] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.439513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1404.000628] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1409.120692] LustreError: 50218:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1435.872406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.454431] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1450.448600] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1617.374450] Lustre: server umount lustre-OST0000 complete [ 1633.497634] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1636.170290] Key type lgssc unregistered [ 1636.523678] LNet: 51896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.534167] LNetError: 51896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.563552] LNet: Removed LNI 192.168.203.149@tcp [ 1637.421339] Key type .llcrypt unregistered [ 1637.425922] Key type ._llcrypt unregistered [ 1664.781483] Key type ._llcrypt registered [ 1664.786343] Key type .llcrypt registered [ 1664.876675] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1665.693201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1665.700071] alg: No test for adler32 (adler32-zlib) [ 1666.862701] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1667.018059] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1668.888494] Key type lgssc registered [ 1670.281593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.818108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.860471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.391902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.663685] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1691.619728] LustreError: 53310:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.653365] LustreError: 53310:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1695.355651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.724826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1699.724507] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1707.293043] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1712.635920] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1718.742400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1719.005731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.657554] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1727.225537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1731.055946] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.335380] Lustre: server umount lustre-OST0000 complete [ 1741.793974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1741.803060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.812025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.822731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.829140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.840276] Lustre: Skipped 1 previous similar message [ 1747.951329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1747.957969] Lustre: Skipped 1 previous similar message [ 1753.056500] Lustre: 52621:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848905/real 1787848905] req@00000000073748fa x1874695380034048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787848921 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1753.058637] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1753.059688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.083703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.080161] Lustre: 52620:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787848906/real 1787848906] req@0000000040c04b23 x1874695380035072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787848922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1755.276564] Lustre: server umount lustre-MDT0000 complete [ 1758.182476] LustreError: 54359:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1758.520894] LustreError: 53290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848927 with bad export cookie 8364707658850895958 [ 1758.528582] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.548689] LustreError: 53290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.814303] Lustre: server umount lustre-MDT0001 complete [ 1769.246380] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1771.488400] Key type lgssc unregistered [ 1771.803768] LNet: 56846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.814766] LNetError: 56846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.832673] LNet: Removed LNI 192.168.203.149@tcp [ 1772.763676] Key type .llcrypt unregistered [ 1772.766257] Key type ._llcrypt unregistered [ 1807.433524] Key type ._llcrypt registered [ 1807.436605] Key type .llcrypt registered [ 1807.503772] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1807.977774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.987529] alg: No test for adler32 (adler32-zlib) [ 1809.009221] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1809.280588] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1810.976178] Key type lgssc registered [ 1812.040134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1819.503760] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1819.522638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.168378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1840.288993] LustreError: 58235:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1866.976380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1873.051325] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1880.873687] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1887.399755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.011588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.091415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1892.624601] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1893.352803] LustreError: 58261:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.365483] LustreError: 58261:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1898.474988] LustreError: 58260:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.540762] LustreError: 59270:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.567311] LustreError: 59270:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1901.649154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.093634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.515506] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1915.000627] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.302283] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.079529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1924.091991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.104486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.661772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.665812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1927.684488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1927.691925] Lustre: Skipped 1 previous similar message [ 1929.857888] Lustre: server umount lustre-OST0000 complete [ 1953.249875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.253769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.291825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.301309] Lustre: Skipped 1 previous similar message [ 1958.369371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.380514] Lustre: Skipped 1 previous similar message [ 1958.602553] Lustre: server umount lustre-MDT0000 complete [ 1962.305221] LustreError: 58242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849131 with bad export cookie 14270796283810950502 [ 1962.325632] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.845326] Lustre: server umount lustre-MDT0001 complete [ 1980.284898] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1983.082860] Key type lgssc unregistered [ 1983.402797] LNet: 61984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.417143] LNetError: 61984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.430188] LNet: Removed LNI 192.168.203.149@tcp [ 1984.177492] Key type .llcrypt unregistered [ 1984.180136] Key type ._llcrypt unregistered [ 2004.159753] Key type ._llcrypt registered [ 2004.168035] Key type .llcrypt registered [ 2004.253594] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2005.135977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2005.261261] alg: No test for adler32 (adler32-zlib) [ 2006.264546] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2006.473687] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2008.112170] Key type lgssc registered [ 2009.298072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.562587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2017.590808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.052949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.607743] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2028.517881] LustreError: 63397:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2028.557593] LustreError: 63397:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2033.641751] LustreError: 63396:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2034.887049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.282823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2039.914129] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2048.554887] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2054.316410] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2061.030146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2061.369894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.428838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2067.946483] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2076.492616] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2095.084494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.092016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2095.112667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.126808] Lustre: Skipped 1 previous similar message [ 2096.455774] Lustre: server umount lustre-OST0000 complete [ 2101.729019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2101.740780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.760321] Lustre: Skipped 1 previous similar message [ 2101.765707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.314199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.318102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.472951] Lustre: server umount lustre-MDT0000 complete [ 2109.783203] LustreError: 63379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849278 with bad export cookie 13304574924275925953 [ 2109.790422] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.802989] LustreError: 63379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2110.091388] Lustre: server umount lustre-MDT0001 complete [ 2121.744056] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2124.690427] Key type lgssc unregistered [ 2125.022784] LNet: 66954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.040632] LNetError: 66954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.065391] LNet: Removed LNI 192.168.203.149@tcp [ 2126.108660] Key type .llcrypt unregistered [ 2126.113133] Key type ._llcrypt unregistered [ 2153.702875] Key type ._llcrypt registered [ 2153.712411] Key type .llcrypt registered [ 2153.906912] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2155.188085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.233220] alg: No test for adler32 (adler32-zlib) [ 2156.247720] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2156.523588] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2158.232160] Key type lgssc registered [ 2159.518265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.133376] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:48:56 (1787849336) [ 2176.180929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.209405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.773799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.307683] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2188.259583] LustreError: 68660:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.287218] LustreError: 68660:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2190.504416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.853888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2195.161934] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2202.711517] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2208.264772] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2215.662386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.048720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.701540] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2224.264664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2230.293695] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2240.271087] Lustre: Failing over lustre-MDT0000 [ 2240.550872] Lustre: server umount lustre-MDT0000 complete [ 2240.995221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.007367] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.017891] Lustre: Skipped 2 previous similar messages [ 2241.042641] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2243.584777] LustreError: 68654:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.606996] LustreError: 68654:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2246.120360] LustreError: 68659:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2246.141824] LustreError: 68659:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2250.205201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.324292] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.454821] Lustre: *** cfs_fail_loc=135, val=0*** [ 2250.577802] LustreError: 71750:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2250.585515] LustreError: 71750:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 2250.595863] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2250.595863] [ 2250.606485] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2250.625196] LustreError: 71738:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2250.648613] LustreError: 71738:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2250.668166] LustreError: 71738:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2250.675729] LustreError: 71738:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2250.718523] Lustre: server umount lustre-MDT0000 complete [ 2250.720764] LustreError: 71738:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2251.236460] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.261646] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2257.879544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.033328] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.048359] LustreError: 72096:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.149@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000008b3d0153) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2258.069887] LustreError: 68651:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2258.369817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.454040] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2258.922432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2263.273142] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2263.540355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2263.576408] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2263.618947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2271.718814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2271.727305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.754102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.764500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2273.772550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.940216] Lustre: server umount lustre-OST0000 complete [ 2284.001676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.003261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.007488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.007497] Lustre: Skipped 1 previous similar message [ 2284.024732] Lustre: Skipped 1 previous similar message [ 2287.010200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.018592] Lustre: Skipped 2 previous similar messages [ 2292.193378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.202131] Lustre: Skipped 1 previous similar message [ 2296.288222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2296.491766] Lustre: server umount lustre-MDT0000 complete [ 2297.314181] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.328650] LustreError: 69667:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 2300.019893] LustreError: 68639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849468 with bad export cookie 6347670875138220883 [ 2300.021285] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.035513] LustreError: 68639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2300.361076] Lustre: server umount lustre-MDT0001 complete [ 2311.427419] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2313.941145] Key type lgssc unregistered [ 2314.175773] LNet: 73536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.183766] LNetError: 73536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2314.200717] LNet: Removed LNI 192.168.203.149@tcp [ 2314.828961] Key type .llcrypt unregistered [ 2314.836650] Key type ._llcrypt unregistered [ 2336.732359] Key type ._llcrypt registered [ 2336.737291] Key type .llcrypt registered [ 2336.901842] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2338.912848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2338.958256] alg: No test for adler32 (adler32-zlib) [ 2340.537616] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2341.025201] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2343.032177] Key type lgssc registered [ 2346.017030] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2357.100695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2357.125544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.852890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.577988] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2368.994874] LustreError: 74950:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.009675] LustreError: 74950:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2374.129700] LustreError: 74951:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.201794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.373153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2382.152634] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2393.841199] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2401.799077] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2408.547470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.917480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.549985] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2423.260373] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2429.434836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2438.440943] Lustre: Failing over lustre-MDT0000 [ 2438.627837] Lustre: server umount lustre-MDT0000 complete [ 2439.170498] LustreError: 74946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.651039] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.659701] Lustre: Skipped 2 previous similar messages [ 2444.258143] LustreError: 74946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.289995] LustreError: 74946:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2447.581049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.726854] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.871223] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2447.989583] LustreError: 78061:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2447.997663] LustreError: 78061:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -9 [ 2448.007338] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2448.007338] [ 2448.016935] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2448.031845] LustreError: 78050:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2448.038092] LustreError: 78050:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2448.044547] LustreError: 78050:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2448.052699] LustreError: 78050:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2448.072774] Lustre: server umount lustre-MDT0000 complete [ 2448.075522] LustreError: 78050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2449.389084] LustreError: 74945:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.412685] LustreError: 74945:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2454.629836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.733131] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.944807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.991075] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2458.943043] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2459.619572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2460.139370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2460.170536] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2460.218491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2467.299723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.313199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.329404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.380242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2470.380645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2472.446104] Lustre: server umount lustre-OST0000 complete [ 2480.609832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.610776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.626839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.634870] Lustre: Skipped 2 previous similar messages [ 2482.487613] Lustre: server umount lustre-MDT0000 complete [ 2485.731430] LustreError: 74950:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.746544] LustreError: 74950:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 2485.912870] LustreError: 74933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849654 with bad export cookie 257796838260876625 [ 2485.915552] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.926463] LustreError: 74933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2486.205804] Lustre: server umount lustre-MDT0001 complete [ 2496.624154] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2499.335042] Key type lgssc unregistered [ 2499.594658] LNet: 79842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.603456] LNetError: 79842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.637178] LNet: Removed LNI 192.168.203.149@tcp [ 2500.521260] Key type .llcrypt unregistered [ 2500.525276] Key type ._llcrypt unregistered [ 2514.409598] Key type ._llcrypt registered [ 2514.412219] Key type .llcrypt registered [ 2514.520468] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2514.914622] Key type .llcrypt unregistered [ 2514.921384] Key type ._llcrypt unregistered [ 2536.608870] Key type ._llcrypt registered [ 2536.614352] Key type .llcrypt registered [ 2536.708288] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2537.718034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.748569] alg: No test for adler32 (adler32-zlib) [ 2538.799283] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2539.051307] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2540.720158] Key type lgssc registered [ 2541.672459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2548.137894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2548.162365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.531223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.215931] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2559.969831] LustreError: 81728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.988449] LustreError: 81728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2560.670570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.051740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2565.325672] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2572.501980] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.743688] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2583.638749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2583.965519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.519918] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2595.254849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2596.502690] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2605.540085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2605.555617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.562664] Lustre: Skipped 1 previous similar message [ 2610.661412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.669313] Lustre: Skipped 1 previous similar message [ 2611.280190] Lustre: server umount lustre-OST0000 complete [ 2615.778395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.784266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.792321] Lustre: Skipped 2 previous similar messages [ 2615.805770] Lustre: Skipped 1 previous similar message [ 2620.899946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.309614] Lustre: server umount lustre-MDT0000 complete [ 2624.580566] LustreError: 82462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849793 with bad export cookie 16861392688840500423 [ 2624.592067] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.597038] LustreError: 82462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2625.058798] Lustre: server umount lustre-MDT0001 complete [ 2635.811775] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2637.983883] Key type lgssc unregistered [ 2638.267133] LNet: 85265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2638.271682] LNetError: 85265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2638.289129] LNet: Removed LNI 192.168.203.149@tcp [ 2639.042943] Key type .llcrypt unregistered [ 2639.045610] Key type ._llcrypt unregistered [ 2659.670418] Key type ._llcrypt registered [ 2659.672524] Key type .llcrypt registered [ 2659.763740] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2660.463233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2660.485563] alg: No test for adler32 (adler32-zlib) [ 2661.534758] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2661.828486] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2663.496775] Key type lgssc registered [ 2664.697693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2671.928890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.972779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.488801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.228310] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2683.880425] LustreError: 86657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.897461] LustreError: 86657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2685.520695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.890060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.989375] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2697.490544] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2703.632585] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2712.060799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2712.252565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2718.465979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2718.875607] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2726.906431] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.098964] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2736.104198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.105549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2738.204871] Lustre: server umount lustre-OST0000 complete [ 2746.341177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.357429] Lustre: Skipped 1 previous similar message [ 2746.372160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.375061] Lustre: Skipped 1 previous similar message [ 2748.781452] Lustre: server umount lustre-MDT0000 complete [ 2751.468915] LustreError: 86652:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2751.482907] LustreError: 86652:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2752.827890] LustreError: 86637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849921 with bad export cookie 2309087951499009587 [ 2752.837028] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2752.846194] LustreError: 86637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.211606] Lustre: server umount lustre-MDT0001 complete [ 2766.792659] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2770.399780] Key type lgssc unregistered [ 2770.775863] LNet: 90192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2770.782450] LNetError: 90192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.816551] LNet: Removed LNI 192.168.203.149@tcp [ 2772.757677] Key type .llcrypt unregistered [ 2772.759660] Key type ._llcrypt unregistered [ 2796.170928] Key type ._llcrypt registered [ 2796.173250] Key type .llcrypt registered [ 2796.261463] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2797.664130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2797.684195] alg: No test for adler32 (adler32-zlib) [ 2798.780286] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2799.036818] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2800.704339] Key type lgssc registered [ 2802.023715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2810.303349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2810.325974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.863390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.614594] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2822.115807] LustreError: 91607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.138060] LustreError: 91607:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2825.671170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.990398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2831.658776] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2840.072467] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2846.449157] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2853.728546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.163746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.273174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2860.283323] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2867.990825] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2883.041125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2883.057120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.061980] LustreError: Skipped 1 previous similar message [ 2883.096881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.113985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2886.116242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.139508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.176385] Lustre: Skipped 1 previous similar message [ 2887.898832] Lustre: server umount lustre-OST0000 complete [ 2892.769686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2892.786954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.819720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.359080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.380390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.398977] Lustre: Skipped 1 previous similar message [ 2898.727274] Lustre: server umount lustre-MDT0000 complete [ 2901.473645] LustreError: 91607:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.495610] LustreError: 91607:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2903.200913] LustreError: 92343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850071 with bad export cookie 8774124399642753320 [ 2903.218402] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.230053] LustreError: 92343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2903.849871] Lustre: server umount lustre-MDT0001 complete [ 2916.450962] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2919.369534] Key type lgssc unregistered [ 2919.759424] LNet: 95183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2919.770081] LNetError: 95183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2919.790917] LNet: Removed LNI 192.168.203.149@tcp [ 2920.800106] Key type .llcrypt unregistered [ 2920.801992] Key type ._llcrypt unregistered [ 2945.188135] Key type ._llcrypt registered [ 2945.190868] Key type .llcrypt registered [ 2945.333640] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2946.872906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2946.922687] alg: No test for adler32 (adler32-zlib) [ 2947.976953] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2948.210973] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2949.905921] Key type lgssc registered [ 2950.908528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.717944] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.749333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2974.368643] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2979.488693] LustreError: 96570:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3006.304234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3014.494229] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3022.731937] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3027.557933] Lustre: server umount lustre-OST0000 complete [ 3035.878686] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:03:23 (1787850203) [ 3042.496906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.928638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.046404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3048.421190] LustreError: 98208:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.438088] LustreError: 98208:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3052.520229] LustreError: 98227:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.084381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3055.612482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.210726] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3067.327888] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3073.448281] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3079.648963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.971466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.504354] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3088.192509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3093.822398] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3108.848824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.855525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.872969] Lustre: Skipped 1 previous similar message [ 3108.893082] Lustre: Skipped 1 previous similar message [ 3113.084744] Lustre: server umount lustre-OST0000 complete [ 3119.085553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.097196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.126858] Lustre: Skipped 1 previous similar message [ 3119.148529] Lustre: Skipped 1 previous similar message [ 3124.196435] LustreError: 98203:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.224234] LustreError: 98203:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3125.953279] Lustre: server umount lustre-MDT0000 complete [ 3128.306585] LustreError: 98209:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.342125] LustreError: 98209:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3130.063635] LustreError: 101152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850298 with bad export cookie 13109574979345215041 [ 3130.071840] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3130.075327] LustreError: 101152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3130.611493] Lustre: server umount lustre-MDT0001 complete [ 3143.590586] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3146.373398] Key type lgssc unregistered [ 3146.634739] LNet: 101906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3146.642142] LNetError: 101906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3146.659014] LNet: Removed LNI 192.168.203.149@tcp [ 3147.529761] Key type .llcrypt unregistered [ 3147.531845] Key type ._llcrypt unregistered [ 3195.990682] Key type ._llcrypt registered [ 3195.993122] Key type .llcrypt registered [ 3196.091765] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3196.819996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3196.843088] alg: No test for adler32 (adler32-zlib) [ 3197.802350] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3197.973076] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3199.650472] Key type lgssc registered [ 3200.634809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3207.971974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3207.987570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.367350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.389773] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3219.940107] LustreError: 104093:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.955959] LustreError: 104093:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3221.056523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.485812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.862300] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3232.406622] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3237.670807] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3244.347583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3244.708740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.805301] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3254.969042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3260.243194] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3272.675280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.685300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.712177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.749727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3275.751321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.752471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.815431] Lustre: Skipped 1 previous similar message [ 3277.101250] Lustre: server umount lustre-OST0000 complete [ 3282.914457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3282.920756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3282.936435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3285.985825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.989862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3286.013439] Lustre: Skipped 1 previous similar message [ 3287.502795] Lustre: server umount lustre-MDT0000 complete [ 3291.092326] LustreError: 104073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850459 with bad export cookie 16168477638958180354 [ 3291.094981] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.109879] LustreError: 104073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3291.736505] Lustre: server umount lustre-MDT0001 complete [ 3304.012527] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3306.909109] Key type lgssc unregistered [ 3307.184629] LNet: 107628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.199983] LNetError: 107628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.217815] LNet: Removed LNI 192.168.203.149@tcp [ 3308.125535] Key type .llcrypt unregistered [ 3308.128831] Key type ._llcrypt unregistered [ 3326.167054] Key type ._llcrypt registered [ 3326.172507] Key type .llcrypt registered [ 3326.326582] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3327.436394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3327.596316] alg: No test for adler32 (adler32-zlib) [ 3328.736441] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3328.965688] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3330.632237] Key type lgssc registered [ 3331.576816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3339.376120] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3339.404742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3355.041681] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3360.161747] LustreError: 108871:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3386.784825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.934795] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3401.006929] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3411.072443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.532154] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3411.549904] LustreError: 109895:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3411.557558] LustreError: 109895:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3411.566820] LustreError: 109895:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3411.732814] Lustre: server umount lustre-MDT0000 complete [ 3411.742495] LustreError: 109895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3430.880745] Lustre: 108209:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850583/real 1787850583] req@00000000e0edfe48 x1874697122745728/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787850599 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3430.924543] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.950489] LustreError: 108883:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3431.057811] Lustre: server umount lustre-OST0000 complete [ 3442.000247] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3452.086145] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3486.379462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.561643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3486.582741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3486.635354] Lustre: lustre-MDT0000: new disk, initializing [ 3486.678412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.702963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3491.344760] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3503.458478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.587024] Lustre: 112686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3503.657419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3503.669817] Lustre: Skipped 1 previous similar message [ 3503.743676] Lustre: lustre-MDT0001: new disk, initializing [ 3503.858489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3503.939067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3503.952451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3509.275133] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3519.344283] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3527.130636] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3536.405759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.994842] Lustre: lustre-OST0000: new disk, initializing [ 3537.019473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3537.039065] Lustre: 114655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3537.234249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.737803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3543.753334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3543.951299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3546.143915] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3557.424235] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3561.740969] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3561.981466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3566.627463] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3566.851787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3569.646217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.649776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.670328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.683410] Lustre: Skipped 1 previous similar message [ 3574.768630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3574.777692] Lustre: Skipped 1 previous similar message [ 3579.875529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3579.893525] Lustre: Skipped 1 previous similar message [ 3582.947259] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3583.220337] Lustre: server umount lustre-OST0000 complete [ 3588.074889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.088809] Lustre: Skipped 2 previous similar messages [ 3588.093715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.108924] Lustre: Skipped 1 previous similar message [ 3594.211935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.220419] Lustre: Skipped 3 previous similar messages [ 3601.376599] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3601.651782] Lustre: server umount lustre-MDT0000 complete [ 3605.460669] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850774 with bad export cookie 10989357954681664585 [ 3605.463451] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.476554] LustreError: 112679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3605.863620] Lustre: server umount lustre-MDT0001 complete [ 3615.429316] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:13:02 (1787850782) [ 3618.474986] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3635.266684] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3645.619811] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3691.561630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.895865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3691.980904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3692.108790] Lustre: lustre-MDT0000: new disk, initializing [ 3692.183804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3692.228105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3698.084335] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3709.746805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.867283] Lustre: 119568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3709.944364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3709.948736] Lustre: Skipped 1 previous similar message [ 3710.067832] Lustre: lustre-MDT0001: new disk, initializing [ 3710.226240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3710.281261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3710.294817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3715.681424] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3726.022585] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3732.665727] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3739.706384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.967586] Lustre: lustre-OST0000: new disk, initializing [ 3739.971422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3739.976920] Lustre: 121538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.084455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3746.275978] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3750.459484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3750.481606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3750.618981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3756.290322] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3761.772779] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3761.974893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3767.025501] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3767.279897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3771.362113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3771.369256] LustreError: Skipped 1 previous similar message [ 3771.372541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.393333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3771.402034] Lustre: Skipped 3 previous similar messages [ 3775.554097] Lustre: server umount lustre-OST0000 complete [ 3781.605150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.621565] Lustre: Skipped 2 previous similar messages [ 3785.638789] Lustre: server umount lustre-MDT0000 complete [ 3786.722833] LustreError: 119576:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3786.732652] LustreError: 119576:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 3789.844559] LustreError: 119560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850958 with bad export cookie 10989357954681665467 [ 3789.844993] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.866592] LustreError: 119560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3790.239162] Lustre: server umount lustre-MDT0001 complete [ 3798.932911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.435814] LustreError: 123754:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.537395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3803.779476] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3804.654727] LustreError: 123755:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.764632] LustreError: 123754:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.928151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.793823] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3826.808827] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3833.708459] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3841.578968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3848.261077] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3855.514409] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3865.660257] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3870.691729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3870.707036] LustreError: Skipped 1 previous similar message [ 3870.713394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3870.729580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3870.732267] Lustre: Skipped 2 previous similar messages [ 3873.762185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3876.023472] Lustre: server umount lustre-OST0000 complete [ 3880.935840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.942032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.950076] Lustre: Skipped 1 previous similar message [ 3885.922217] Lustre: server umount lustre-MDT0000 complete [ 3888.033954] LustreError: 123750:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.047699] LustreError: 123750:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3889.697280] LustreError: 125601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851058 with bad export cookie 10989357954681666342 [ 3889.699668] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3889.705542] LustreError: 125601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3890.162947] Lustre: server umount lustre-MDT0001 complete [ 3902.640900] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3905.674770] Key type lgssc unregistered [ 3906.129129] LNet: 127388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3906.137579] LNetError: 127388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3906.154407] LNet: Removed LNI 192.168.203.149@tcp [ 3907.227491] Key type .llcrypt unregistered [ 3907.233035] Key type ._llcrypt unregistered [ 3925.914367] Key type ._llcrypt registered [ 3925.916677] Key type .llcrypt registered [ 3926.006412] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 3942.809572] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3943.938629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3944.056942] alg: No test for adler32 (adler32-zlib) [ 3945.096248] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 3945.300318] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3946.928161] Key type lgssc registered [ 3947.969582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3985.496996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3985.529760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.806751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3986.838283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3986.943841] Lustre: lustre-MDT0000: new disk, initializing [ 3987.082522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.106259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3992.314516] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4003.453944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.537530] Lustre: 130543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4003.579680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4003.583322] Lustre: Skipped 1 previous similar message [ 4003.668294] Lustre: lustre-MDT0001: new disk, initializing [ 4003.810239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4003.890587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4003.903312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4008.638422] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4018.068188] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.885669] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4031.373670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4031.771234] Lustre: lustre-OST0000: new disk, initializing [ 4031.779116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4031.790918] Lustre: 132510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4031.901072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.139086] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4041.242666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4041.250837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4041.367713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4048.948483] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.714346] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4053.035736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4058.561971] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4058.826589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4061.665692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4061.678389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.688598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.193923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4062.202157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4062.222486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.835122] Lustre: server umount lustre-OST0000 complete [ 4072.417511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.433220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4072.446286] Lustre: Skipped 2 previous similar messages [ 4077.538624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.545763] Lustre: Skipped 1 previous similar message [ 4077.931810] Lustre: server umount lustre-MDT0000 complete [ 4081.542914] LustreError: 130534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851250 with bad export cookie 4738983446116149955 [ 4081.556369] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4081.564934] LustreError: 130534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4081.944657] Lustre: server umount lustre-MDT0001 complete [ 4091.568184] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:20:58 (1787851258) [ 4100.969547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.647818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.723896] LustreError: 134924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.758258] LustreError: 134924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4107.470404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4111.842772] LustreError: 134925:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.973126] LustreError: 134924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4117.615927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4118.137650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4123.313068] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4132.044663] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4139.424058] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.403806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.405392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.424779] Lustre: Skipped 2 previous similar messages [ 4147.691753] LustreError: 136358:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.717553] LustreError: 136358:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4147.983822] Lustre: server umount lustre-MDT0000 complete [ 4152.805581] LustreError: 134924:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.862849] LustreError: 134924:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4153.465192] LustreError: 134904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851322 with bad export cookie 4738983446116150830 [ 4153.469026] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.476623] LustreError: 134904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4153.775416] Lustre: server umount lustre-MDT0001 complete [ 4165.869093] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:22:12 (1787851332) [ 4173.070465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.576962] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4193.696440] LustreError: 137343:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4219.424502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.609868] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4235.273694] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4237.761663] Lustre: server umount lustre-OST0000 complete [ 4247.390250] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:23:34 (1787851414) [ 4255.133285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4255.680924] LustreError: 138789:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4255.798724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.833665] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4269.939367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.080300] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4284.097170] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4291.089715] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4297.659586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.072390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4298.078778] Lustre: Skipped 1 previous similar message [ 4305.315372] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4313.531235] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4325.860361] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.868725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.884016] Lustre: Skipped 2 previous similar messages [ 4325.884303] Lustre: Skipped 4 previous similar messages [ 4326.677858] Lustre: server umount lustre-MDT0000 complete [ 4330.593169] LustreError: 139525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851499 with bad export cookie 4738983446116151502 [ 4330.602399] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4330.609583] LustreError: 139525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4330.958560] Lustre: server umount lustre-MDT0001 complete [ 4345.289265] Lustre: server umount lustre-OST0000 complete [ 4353.753704] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:25:20 (1787851520) [ 4360.551264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.969699] LustreError: 142442:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.993840] LustreError: 142442:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4365.538785] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4374.068987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.591768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4374.601637] Lustre: Skipped 1 previous similar message [ 4379.226903] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4387.215945] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4394.346827] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4401.508332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.945429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4408.466054] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4416.985382] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4422.150297] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4422.320877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4426.782219] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4427.003307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4429.796846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.834435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4429.845185] Lustre: Skipped 2 previous similar messages [ 4431.111106] Lustre: server umount lustre-OST0000 complete [ 4449.248185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4449.646739] Lustre: server umount lustre-MDT0000 complete [ 4454.373103] LustreError: 142438:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4454.401595] LustreError: 142438:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4455.069866] LustreError: 142424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851623 with bad export cookie 4738983446116152566 [ 4455.082693] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.105069] LustreError: 142424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4455.673394] Lustre: server umount lustre-MDT0001 complete [ 4468.765119] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:27:15 (1787851635) [ 4476.230293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4491.809107] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4496.928638] LustreError: 146602:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4522.593552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.596040] Lustre: Skipped 1 previous similar message [ 4530.152966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4539.656221] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4548.454087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.086713] LustreError: 146627:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.307887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4554.713236] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4563.834246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.073794] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4578.597965] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4586.423841] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4592.446662] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4592.674499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4597.631256] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4597.873276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4604.900282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4604.908954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.919554] Lustre: Skipped 3 previous similar messages [ 4604.924971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4604.927649] Lustre: Skipped 8 previous similar messages [ 4606.863890] Lustre: server umount lustre-MDT0000 complete [ 4611.205530] LustreError: 146610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851779 with bad export cookie 4738983446116153595 [ 4611.211127] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4611.212183] LustreError: 146610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4611.427900] Lustre: server umount lustre-MDT0001 complete [ 4626.384492] Lustre: server umount lustre-OST0000 complete [ 4637.352735] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:30:04 (1787851804) [ 4645.294111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.154843] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4666.272766] LustreError: 150803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 4698.321633] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4706.921711] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4713.839054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.242393] LustreError: 150828:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4714.258759] LustreError: 150828:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 4714.342666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4719.388284] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4729.402123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.423332] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4744.229304] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4751.501923] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4759.471025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4759.727655] Lustre: lustre-OST0001: new disk, initializing [ 4759.732828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4759.747285] Lustre: 153676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4767.911895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4767.952200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4767.966367] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4768.177358] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4779.180190] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4785.238577] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4785.476594] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4790.455733] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4790.680231] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4793.856901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.869113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4793.880747] Lustre: Skipped 3 previous similar messages [ 4793.894328] Lustre: Skipped 3 previous similar messages [ 4799.289994] Lustre: server umount lustre-OST0000 complete [ 4804.076103] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4804.099024] LustreError: Skipped 1 previous similar message [ 4809.414632] Lustre: server umount lustre-OST0001 complete [ 4819.162417] Lustre: server umount lustre-MDT0000 complete [ 4823.408236] LustreError: 153674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851992 with bad export cookie 4738983446116154547 [ 4823.414702] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4823.430811] LustreError: 153674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4824.017228] Lustre: server umount lustre-MDT0001 complete [ 4850.023414] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:33:37 (1787852017) [ 4851.807793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4853.481152] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4855.934626] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:33:42 (1787852022) [ 4864.487340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.992615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4865.034778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4865.349287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.371601] Lustre: Skipped 6 previous similar messages [ 4869.867346] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4879.234132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.333180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4879.360203] Lustre: 157236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4884.478643] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4892.388794] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4898.665366] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4905.405674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4905.575582] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4913.601812] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4913.934157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4923.815707] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4929.464413] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4929.955477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4934.248962] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4934.408598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4939.759041] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.764390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.767400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.767409] Lustre: Skipped 9 previous similar messages [ 4939.771816] LustreError: Skipped 1 previous similar message [ 4939.803383] Lustre: Skipped 5 previous similar messages [ 4942.445382] Lustre: server umount lustre-OST0000 complete [ 4968.945739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.317591] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4979.661836] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.716287] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4980.726424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.752744] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4992.141424] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4999.133129] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4999.357089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5004.888932] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5005.169299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5011.649873] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9231c7c8e800.ost_server_uuid 50 [ 5013.235318] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9231c7c8e800.ost_server_uuid in FULL state after 0 sec [ 5013.457435] LustreError: 157243:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.49@tcp) returned error from blocking AST (req@00000000d46eacd6 x1874697769988352 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006520834b/0x41c440560635c60d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xea5e7998b65964cd expref: 5 pid: 159109 timeout: 5113 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5013.486543] LustreError: lustre-MDT0000: A client on nid 192.168.203.49@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5013.494645] LustreError: 157232:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.49@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006520834b/0x41c440560635c60d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0xea5e7998b65964cd expref: 6 pid: 159109 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5019.111180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5019.876246] Lustre: server umount lustre-OST0000 complete [ 5029.508304] Lustre: server umount lustre-MDT0000 complete [ 5031.905587] LustreError: 157248:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.952317] LustreError: 157248:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 11 previous similar messages [ 5033.490660] LustreError: 157980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852202 with bad export cookie 4738983446116155723 [ 5033.492165] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5033.513836] LustreError: 157980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5044.057677] Lustre: server umount lustre-MDT0001 complete [ 5059.177624] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5062.783739] Key type lgssc unregistered [ 5063.100299] LNet: 163360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5063.111967] LNetError: 163360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5063.130206] LNet: Removed LNI 192.168.203.149@tcp [ 5064.485459] Key type .llcrypt unregistered [ 5064.488451] Key type ._llcrypt unregistered [ 5089.167484] Key type ._llcrypt registered [ 5089.170434] Key type .llcrypt registered [ 5089.299153] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5090.370995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5090.393358] alg: No test for adler32 (adler32-zlib) [ 5091.407693] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5091.623619] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5093.360151] Key type lgssc registered [ 5094.975057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5104.315164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5104.335540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.885087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5111.691479] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5115.364819] LustreError: 164771:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.396503] LustreError: 164771:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5120.489821] LustreError: 164772:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.135689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.407229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.627777] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5137.219373] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5144.461573] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5153.171438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5153.677870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5160.290575] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5169.389831] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5175.297443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5180.935262] Lustre: Failing over lustre-MDT0000 [ 5181.361517] Lustre: server umount lustre-MDT0000 complete [ 5181.436600] LustreError: 164766:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.993239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5185.004052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.026384] LustreError: 165500:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.514484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.542317] Lustre: Skipped 1 previous similar message [ 5190.626202] LustreError: 164768:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.664206] LustreError: 164768:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 5193.403236] LustreError: 164751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.49@tcp arrived at 1787852362 with bad export cookie 303367599496554823 [ 5200.753387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.872250] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.885428] LustreError: 164080:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008563ae21 x1874698971415936/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5200.911290] LustreError: 164767:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.928922] LustreError: 164767:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 5200.930209] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x435c739745aea90 [ 5200.943319] Lustre: MGC192.168.203.149@tcp: Connection restored to 0@lo (at 0@lo) [ 5201.207393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.291645] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5206.276209] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5206.503837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5206.512391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5208.687902] Lustre: lustre-MDT0000: Denying connection for new client 306b76c2-350b-4a29-bbb2-f55453a2e74b (at 192.168.203.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5220.322936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5220.352260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.373638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.867874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.880808] Lustre: Skipped 1 previous similar message [ 5226.980169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5226.989446] Lustre: Skipped 1 previous similar message [ 5231.073136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5231.348411] Lustre: server umount lustre-OST0000 complete [ 5235.756273] LustreError: 168506:0:(ldlm_lib.c:3039:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5235.776756] Lustre: 167853:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5235.800857] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5235.825426] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5235.840269] LustreError: 167853:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d343a82f x1874698971433344/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5235.859365] LustreError: 167853:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5235.869234] LustreError: 167853:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5235.888458] Lustre: lustre-MDT0000: Recovery over after 0:29, of 2 clients 0 recovered and 2 were evicted. [ 5236.235855] Lustre: server umount lustre-MDT0000 complete [ 5237.223105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.258403] Lustre: Skipped 1 previous similar message [ 5237.275271] LustreError: 164766:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5241.569565] LustreError: 164752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852410 with bad export cookie 303367599496555152 [ 5241.585131] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5241.590250] LustreError: 164752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5242.068025] Lustre: server umount lustre-MDT0001 complete [ 5255.960311] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5258.817412] Key type lgssc unregistered [ 5259.174548] LNet: 169264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5259.186199] LNetError: 169264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5259.213420] LNet: Removed LNI 192.168.203.149@tcp [ 5260.111910] Key type .llcrypt unregistered [ 5260.121270] Key type ._llcrypt unregistered [ 5293.146822] Key type ._llcrypt registered [ 5293.149430] Key type .llcrypt registered [ 5293.246742] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5294.451407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5294.543747] alg: No test for adler32 (adler32-zlib) [ 5295.528092] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5295.826863] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5297.592168] Key type lgssc registered [ 5298.950829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5307.760919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5307.826463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5309.070342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.993388] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5318.645574] LustreError: 170677:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5318.673326] LustreError: 170677:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5323.730734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5323.746562] LustreError: 170678:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.107771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5328.741714] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5338.904748] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5348.556721] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5357.787089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.539148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.413063] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5375.832759] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5379.011873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5389.281386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5389.290528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5389.303408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5389.305716] Lustre: Skipped 1 previous similar message [ 5389.320525] Lustre: Skipped 1 previous similar message [ 5392.552908] Lustre: server umount lustre-OST0000 complete [ 5399.526779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.529199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.547358] Lustre: Skipped 1 previous similar message [ 5399.562253] Lustre: Skipped 1 previous similar message [ 5402.872773] Lustre: server umount lustre-MDT0000 complete [ 5404.641264] LustreError: 171683:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.680660] LustreError: 171683:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5406.970291] LustreError: 170657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852575 with bad export cookie 13424442894524557048 [ 5406.982206] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5406.990424] LustreError: 170657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5407.331099] Lustre: server umount lustre-MDT0001 complete [ 5418.326499] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5421.351572] Key type lgssc unregistered [ 5421.659963] LNet: 174217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5421.672163] LNetError: 174217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5421.681567] LNet: Removed LNI 192.168.203.149@tcp [ 5422.465698] Key type .llcrypt unregistered [ 5422.468901] Key type ._llcrypt unregistered [ 5464.614343] Key type ._llcrypt registered [ 5464.616257] Key type .llcrypt registered [ 5464.730657] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5465.855817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5465.885720] alg: No test for adler32 (adler32-zlib) [ 5467.014627] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5467.360790] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5469.009072] Key type lgssc registered [ 5470.568278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5481.120872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5481.164228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.111310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5488.436738] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5492.716723] LustreError: 175930:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.750050] LustreError: 175930:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5497.830739] LustreError: 175931:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5501.426596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.070597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5506.802929] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5515.278843] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5523.852225] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5532.884594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5533.151022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.605477] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5547.543183] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5553.652376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5558.757481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5558.760753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5558.761444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5558.773515] Lustre: Skipped 1 previous similar message [ 5558.794960] Lustre: Skipped 1 previous similar message [ 5560.602239] Lustre: server umount lustre-OST0000 complete [ 5568.484863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5568.497540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.514946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5571.824179] Lustre: server umount lustre-MDT0000 complete [ 5574.130347] LustreError: 175931:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.160504] LustreError: 175931:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5575.962073] LustreError: 175911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852744 with bad export cookie 4103111121649489407 [ 5575.966115] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.984165] LustreError: 175911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5576.568802] Lustre: server umount lustre-MDT0001 complete [ 5590.080118] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5593.213653] Key type lgssc unregistered [ 5593.453931] LNet: 179470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5593.460579] LNetError: 179470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5594.539790] LNet: Removed LNI 192.168.203.149@tcp [ 5595.762555] Key type .llcrypt unregistered [ 5595.766155] Key type ._llcrypt unregistered [ 5622.372197] Key type ._llcrypt registered [ 5622.373996] Key type .llcrypt registered [ 5622.489465] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5623.736435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5623.765442] alg: No test for adler32 (adler32-zlib) [ 5624.887846] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5625.253683] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5626.912199] Key type lgssc registered [ 5628.297680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5640.927623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5640.945461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5642.457490] Lustre: *** cfs_fail_loc=135, val=0*** [ 5642.577701] LustreError: 180995:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5642.583815] LustreError: 180995:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 5642.589158] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5642.589158] [ 5642.594840] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5642.610382] LustreError: 180956:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5642.624914] LustreError: 180956:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5642.636735] LustreError: 180956:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5642.659015] LustreError: 180956:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5642.807428] Lustre: server umount lustre-MDT0000 complete [ 5642.810534] LustreError: 180956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5651.019257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.533220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.962960] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5661.667254] LustreError: 181369:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.713436] LustreError: 181369:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5666.800598] LustreError: 181368:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.866485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.366637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.286956] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5682.434615] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5689.150249] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5694.696571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.702457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5694.719762] Lustre: Skipped 1 previous similar message [ 5697.905707] Lustre: server umount lustre-MDT0000 complete [ 5700.099293] LustreError: 181368:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.109489] LustreError: 181368:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5702.672843] LustreError: 181349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852871 with bad export cookie 3755348228247942919 [ 5702.675022] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5702.681705] LustreError: 181349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5703.068783] Lustre: server umount lustre-MDT0001 complete [ 5719.271934] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5724.160358] Key type lgssc unregistered [ 5724.738732] LNet: 183694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5724.748746] LNetError: 183694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5724.780854] LNet: Removed LNI 192.168.203.149@tcp [ 5726.741325] Key type .llcrypt unregistered [ 5726.744697] Key type ._llcrypt unregistered [ 5754.587197] Key type ._llcrypt registered [ 5754.588995] Key type .llcrypt registered [ 5754.743895] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5755.079406] Key type .llcrypt unregistered [ 5755.083464] Key type ._llcrypt unregistered [ 5776.259868] Key type ._llcrypt registered [ 5776.263035] Key type .llcrypt registered [ 5776.427926] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5777.540944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5777.592473] alg: No test for adler32 (adler32-zlib) [ 5778.575870] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5778.761232] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5780.379696] Key type lgssc registered [ 5781.657321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5790.239771] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5790.262241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.923682] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5811.044578] LustreError: 185834:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 5837.728282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.963088] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5851.609939] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5858.511550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.629969] LustreError: 185170:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000073624fbe x1874699691755776/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5858.964862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5859.026237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5862.919506] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5864.422740] LustreError: 185860:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.438083] LustreError: 185860:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5869.539428] LustreError: 185861:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.797500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.152227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5875.247500] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5882.226799] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5887.307041] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5890.304184] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5903.848815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.858277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5903.860460] Lustre: Skipped 1 previous similar message [ 5903.883200] Lustre: Skipped 1 previous similar message [ 5908.975116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5908.995061] Lustre: Skipped 1 previous similar message [ 5914.086050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.092518] Lustre: Skipped 1 previous similar message [ 5918.177525] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5918.395950] Lustre: server umount lustre-OST0000 complete [ 5922.273168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5922.281038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.289789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5924.326955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.430149] Lustre: server umount lustre-MDT0000 complete [ 5929.457864] LustreError: 186878:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5929.493298] LustreError: 186878:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5932.281347] LustreError: 185843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853101 with bad export cookie 1999853614902383516 [ 5932.286596] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.298215] LustreError: 185843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5932.673177] Lustre: server umount lustre-MDT0001 complete [ 5943.271546] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5946.200366] Key type lgssc unregistered [ 5946.532811] LNet: 189853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5946.557167] LNetError: 189853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5946.591041] LNet: Removed LNI 192.168.203.149@tcp [ 5947.562297] Key type .llcrypt unregistered [ 5947.569409] Key type ._llcrypt unregistered [ 5969.530947] Key type ._llcrypt registered [ 5969.532754] Key type .llcrypt registered [ 5969.634792] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5970.407881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.449660] alg: No test for adler32 (adler32-zlib) [ 5971.595394] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 5971.830521] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5973.536166] Key type lgssc registered [ 5974.984699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5982.246449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5982.289325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.804132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.852709] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5993.960948] LustreError: 191246:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.997894] LustreError: 191246:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5996.870754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.330477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.377415] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6011.772430] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6017.506276] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6024.212712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.605688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.729977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6031.124687] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6039.732351] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6045.977856] Lustre: Failing over lustre-MDT0000 [ 6046.508480] Lustre: server umount lustre-MDT0000 complete [ 6047.747676] LustreError: 191240:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.263769] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.275230] Lustre: Skipped 2 previous similar messages [ 6052.835687] LustreError: 191242:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.861589] LustreError: 191242:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6057.957925] LustreError: 192398:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.990209] LustreError: 192398:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6063.078527] LustreError: 191241:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.099627] LustreError: 191241:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6063.585384] Lustre: 190559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853216/real 1787853216] req@000000002f03244a x1874699894154880/t0(0) o400->MGC192.168.203.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787853232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6063.625570] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6067.820793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6073.401151] LustreError: 191241:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.434958] LustreError: 191241:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 6073.875087] LustreError: 190555:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fc9e3bea x1874699894160000/t0(0) o250->MGC192.168.203.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6074.421394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6074.537178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6078.437759] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6078.471320] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6079.474793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6079.501812] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6079.528130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6084.321087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6091.730509] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6097.899053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6097.915177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.931966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6099.425264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6099.428876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.437733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6103.174532] Lustre: server umount lustre-OST0000 complete [ 6110.181090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.191635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6110.204981] Lustre: Skipped 1 previous similar message [ 6110.220991] Lustre: Skipped 1 previous similar message [ 6113.173160] Lustre: server umount lustre-MDT0000 complete [ 6115.298838] LustreError: 191245:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.320202] LustreError: 191245:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 6117.222950] LustreError: 191980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853285 with bad export cookie 6750291501569294300 [ 6117.227054] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6117.255381] LustreError: 191980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6117.540146] Lustre: server umount lustre-MDT0001 complete [ 6129.228071] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6131.921637] Key type lgssc unregistered [ 6132.252587] LNet: 196091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6132.257693] LNetError: 196091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6132.268378] LNet: Removed LNI 192.168.203.149@tcp [ 6133.131833] Key type .llcrypt unregistered [ 6133.134574] Key type ._llcrypt unregistered [ 6158.092533] Key type ._llcrypt registered [ 6158.097092] Key type .llcrypt registered [ 6158.218765] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6159.249102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6159.264280] alg: No test for adler32 (adler32-zlib) [ 6160.393640] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6160.713740] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6162.448382] Key type lgssc registered [ 6163.744497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6171.330598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6171.380916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.975053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.386445] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6182.373162] LustreError: 197506:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.389846] LustreError: 197506:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6185.927426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.410404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.748405] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6197.919459] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6204.151484] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6211.329178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6211.707364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6217.474260] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6217.850675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6224.837600] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6227.830849] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6232.227535] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6245.860153] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6254.057465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.067063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6254.078234] Lustre: Skipped 1 previous similar message [ 6259.174794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6259.194254] Lustre: Skipped 2 previous similar messages [ 6260.091584] Lustre: server umount lustre-OST0000 complete [ 6264.289458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6264.295448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6264.306371] Lustre: Skipped 1 previous similar message [ 6264.323381] Lustre: Skipped 1 previous similar message [ 6269.410606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6269.414541] Lustre: Skipped 1 previous similar message [ 6270.495899] Lustre: server umount lustre-MDT0000 complete [ 6273.709252] LustreError: 197488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853442 with bad export cookie 11596072317017178987 [ 6273.715743] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6273.720421] LustreError: 197488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6274.059424] Lustre: server umount lustre-MDT0001 complete [ 6284.950721] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6287.553936] Key type lgssc unregistered [ 6287.802361] LNet: 201360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6287.811314] LNetError: 201360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6287.829280] LNet: Removed LNI 192.168.203.149@tcp [ 6288.714286] Key type .llcrypt unregistered [ 6288.716393] Key type ._llcrypt unregistered [ 6307.243419] Key type ._llcrypt registered [ 6307.247207] Key type .llcrypt registered [ 6307.350908] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6308.386523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6308.510645] alg: No test for adler32 (adler32-zlib) [ 6309.581945] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6309.916276] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6311.648160] Key type lgssc registered [ 6313.205653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6321.185757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6321.219431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.651448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.229505] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6332.898023] LustreError: 202626:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.917275] LustreError: 202626:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6334.759382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.064080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.686284] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6346.924498] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.306579] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.638851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.004412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.588840] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6369.266595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6372.243271] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6377.563721] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6389.960742] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6403.066218] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6410.285151] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6421.474574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6421.489817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6421.514525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.572992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.574873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.597100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.691493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.705684] Lustre: Skipped 1 previous similar message [ 6435.296156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6435.461545] Lustre: server umount lustre-OST0000 complete [ 6440.930049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6440.932046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6440.956513] Lustre: Skipped 1 previous similar message [ 6446.049182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.052576] Lustre: Skipped 2 previous similar messages [ 6453.216506] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6453.468030] Lustre: server umount lustre-MDT0000 complete [ 6456.300265] LustreError: 202622:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6456.318401] LustreError: 202622:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6457.032810] LustreError: 202607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853625 with bad export cookie 6258504746468716936 [ 6457.037776] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6457.048673] LustreError: 202607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6457.450699] Lustre: server umount lustre-MDT0001 complete [ 6467.654382] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6470.536879] Key type lgssc unregistered [ 6470.852636] LNet: 207747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6470.863098] LNetError: 207747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.885126] LNet: Removed LNI 192.168.203.149@tcp [ 6471.823265] Key type .llcrypt unregistered [ 6471.826233] Key type ._llcrypt unregistered [ 6493.806660] Key type ._llcrypt registered [ 6493.808900] Key type .llcrypt registered [ 6493.901872] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6495.066091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.075690] alg: No test for adler32 (adler32-zlib) [ 6496.155258] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6496.437198] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6498.136640] Key type lgssc registered [ 6499.482110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.450783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.477771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.947522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.023634] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6518.243417] LustreError: 209169:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.273123] LustreError: 209169:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6520.839744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6521.271374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6525.509240] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6532.476937] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6538.074853] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6544.902737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6545.268674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.655368] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6553.477170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6559.465169] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6574.050570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.053542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.061142] Lustre: Skipped 1 previous similar message [ 6574.062715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6574.083354] Lustre: Skipped 1 previous similar message [ 6579.170721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.187129] Lustre: Skipped 1 previous similar message [ 6579.920978] Lustre: server umount lustre-OST0000 complete [ 6584.290891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.307209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6584.320988] Lustre: Skipped 1 previous similar message [ 6589.411876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.419462] Lustre: Skipped 1 previous similar message [ 6589.902853] Lustre: server umount lustre-MDT0000 complete [ 6593.760467] LustreError: 209149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853762 with bad export cookie 9056381696606680287 [ 6593.762839] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.774191] LustreError: 209149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6594.287795] Lustre: server umount lustre-MDT0001 complete [ 6606.192491] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6609.380928] Key type lgssc unregistered [ 6609.658579] LNet: 212856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6609.666524] LNetError: 212856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6609.680927] LNet: Removed LNI 192.168.203.149@tcp [ 6610.715515] Key type .llcrypt unregistered [ 6610.720374] Key type ._llcrypt unregistered [ 6629.228498] Key type ._llcrypt registered [ 6629.230952] Key type .llcrypt registered [ 6629.357374] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6630.221276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6630.368066] alg: No test for adler32 (adler32-zlib) [ 6631.337492] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6631.540978] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6633.240173] Key type lgssc registered [ 6634.158891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6641.907618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6641.966537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.530375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.919799] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6653.927285] LustreError: 214126:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.950706] LustreError: 214126:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6656.426206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.806178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.383487] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6669.038543] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6673.989968] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6680.568609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.984194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.149737] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6689.207373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6694.945511] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6713.827631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6713.835481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.860778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6714.862978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6714.864885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6714.877991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6716.240448] Lustre: server umount lustre-OST0000 complete [ 6723.554493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6723.561567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6723.582917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6723.602416] Lustre: Skipped 1 previous similar message [ 6726.732695] Lustre: server umount lustre-MDT0000 complete [ 6730.210845] LustreError: 216085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.245508] LustreError: 216085:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6730.318345] LustreError: 214105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853899 with bad export cookie 12181716226899430057 [ 6730.327179] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6730.330132] LustreError: 214105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6730.696814] Lustre: server umount lustre-MDT0001 complete [ 6740.707658] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6743.437744] Key type lgssc unregistered [ 6743.692555] LNet: 217810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6743.696933] LNetError: 217810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6743.719326] LNet: Removed LNI 192.168.203.149@tcp [ 6744.456708] Key type .llcrypt unregistered [ 6744.460874] Key type ._llcrypt unregistered [ 6761.562656] Key type ._llcrypt registered [ 6761.566288] Key type .llcrypt registered [ 6761.651532] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6762.425712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6762.522506] alg: No test for adler32 (adler32-zlib) [ 6763.537440] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6763.762894] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6765.433143] Key type lgssc registered [ 6766.357814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6772.764456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6772.786375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.179502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.310934] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6783.460484] LustreError: 219074:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.483725] LustreError: 219074:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6786.408844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.735305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.007679] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6798.654662] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6803.727133] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6809.938545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6810.261975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6815.087206] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6818.486287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6821.468352] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6829.784494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6829.857979] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6830.079365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.683309] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6842.609468] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6844.320643] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6855.935162] Lustre: Permanently deactivating lustre-OST0001 [ 6855.957325] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6865.496331] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6874.638506] Lustre: Permanently reactivating lustre-OST0001 [ 6874.650955] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6874.655372] Lustre: Skipped 2 previous similar messages [ 6880.909823] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6880.937534] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6880.950845] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6880.968417] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6881.012440] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6891.489215] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6891.498890] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.500241] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6891.511585] Lustre: Skipped 2 previous similar messages [ 6891.519023] Lustre: Skipped 1 previous similar message [ 6894.060571] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6894.071771] Lustre: Skipped 1 previous similar message [ 6899.178677] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6899.190753] Lustre: Skipped 1 previous similar message [ 6903.264139] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6903.332393] Lustre: server umount lustre-OST0001 complete [ 6904.298831] LustreError: 220899:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6904.314319] LustreError: 220899:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6908.898310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6908.903126] LustreError: Skipped 1 previous similar message [ 6908.909918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.921997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6909.415062] LustreError: 220899:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6909.440390] LustreError: 220899:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6912.524210] Lustre: server umount lustre-OST0000 complete [ 6919.650856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.667456] Lustre: Skipped 1 previous similar message [ 6919.674318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6919.679034] Lustre: Skipped 3 previous similar messages [ 6922.060481] Lustre: server umount lustre-MDT0000 complete [ 6924.773651] LustreError: 219070:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.788664] LustreError: 219070:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6925.334837] LustreError: 219056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854094 with bad export cookie 12177786595084340249 [ 6925.341259] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.342937] LustreError: 219056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6925.683289] Lustre: server umount lustre-MDT0001 complete [ 6934.471109] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 6936.749128] Key type lgssc unregistered [ 6937.030478] LNet: 224199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6937.046065] LNetError: 224199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6937.063576] LNet: Removed LNI 192.168.203.149@tcp [ 6937.799225] Key type .llcrypt unregistered [ 6937.801105] Key type ._llcrypt unregistered [ 6974.652687] Key type ._llcrypt registered [ 6974.654593] Key type .llcrypt registered [ 6974.780852] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 6975.897436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6975.914327] alg: No test for adler32 (adler32-zlib) [ 6976.985622] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 6977.217409] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 6978.920206] Key type lgssc registered [ 6979.912972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6985.637731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6985.656215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6986.922160] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6986.937530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6987.057086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.372327] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 6997.711461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6997.778870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6997.795655] Lustre: 226401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6998.044767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7001.343574] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7007.828398] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7012.618708] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7019.109792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7019.334219] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7019.546907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.776644] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7027.814343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7032.268895] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.913664] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7039.210850] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7044.822271] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7055.105841] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7065.362371] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7085.632277] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7085.637389] Lustre: Skipped 2 previous similar messages [ 7122.693947] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7122.704377] Lustre: Skipped 4 previous similar messages [ 7190.519067] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7190.526635] Lustre: Skipped 8 previous similar messages [ 7197.155088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7197.156822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7197.173113] Lustre: Skipped 1 previous similar message [ 7197.189553] Lustre: Skipped 1 previous similar message [ 7202.276509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.287631] Lustre: Skipped 1 previous similar message [ 7207.396514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7207.404173] Lustre: Skipped 1 previous similar message [ 7210.977731] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7211.288582] Lustre: server umount lustre-OST0000 complete [ 7217.641144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7217.650651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7217.662066] Lustre: Skipped 1 previous similar message [ 7217.680510] Lustre: Skipped 1 previous similar message [ 7220.783896] Lustre: server umount lustre-MDT0000 complete [ 7224.334585] LustreError: 227143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854393 with bad export cookie 14770976568581863702 [ 7224.340890] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7224.346265] LustreError: 227143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7224.712571] Lustre: server umount lustre-MDT0001 complete [ 7233.120514] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7235.417507] Key type lgssc unregistered [ 7235.676658] LNet: 231187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7235.693102] LNetError: 231187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7235.710207] LNet: Removed LNI 192.168.203.149@tcp [ 7236.623311] Key type .llcrypt unregistered [ 7236.624607] Key type ._llcrypt unregistered [ 7258.699671] Key type ._llcrypt registered [ 7258.705268] Key type .llcrypt registered [ 7258.813393] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7260.229380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7260.271705] alg: No test for adler32 (adler32-zlib) [ 7261.421718] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7261.730103] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7263.456179] Key type lgssc registered [ 7264.602811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7272.883974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7272.905258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.443076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7279.116063] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7284.706675] LustreError: 232580:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7284.718146] LustreError: 232580:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7288.429389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.885881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7293.083371] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7300.875161] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7306.932804] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7313.566704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7313.963674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.112972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7320.944236] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 7328.736280] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7345.633884] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7345.640395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7345.655104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7347.899828] Lustre: server umount lustre-OST0000 complete [ 7353.832320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.844241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7353.850897] Lustre: Skipped 1 previous similar message [ 7353.871804] Lustre: Skipped 1 previous similar message [ 7357.981649] Lustre: server umount lustre-MDT0000 complete [ 7358.959418] LustreError: 233600:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7358.989255] LustreError: 233600:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7362.389185] LustreError: 232559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854531 with bad export cookie 4967307589014694775 [ 7362.398464] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7372.925444] Lustre: server umount lustre-MDT0001 complete [ 7383.476540] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7386.186392] Key type lgssc unregistered [ 7386.460631] LNet: 236272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7386.467340] LNetError: 236272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7386.482831] LNet: Removed LNI 192.168.203.149@tcp [ 7387.326938] Key type .llcrypt unregistered [ 7387.329804] Key type ._llcrypt unregistered [ 7411.284108] Key type ._llcrypt registered [ 7411.287690] Key type .llcrypt registered [ 7411.412979] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7412.629820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7412.809829] alg: No test for adler32 (adler32-zlib) [ 7413.888751] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7414.133252] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7415.808180] Key type lgssc registered [ 7416.898291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7689.412949] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7692.285392] Key type lgssc unregistered [ 7692.650120] LNet: 238301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7692.661618] LNetError: 238301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7692.683057] LNet: Removed LNI 192.168.203.149@tcp [ 7693.558664] Key type .llcrypt unregistered [ 7693.561156] Key type ._llcrypt unregistered [ 7721.819161] Key type ._llcrypt registered [ 7721.820770] Key type .llcrypt registered [ 7721.967982] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 7738.863396] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 7739.954282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7740.007115] alg: No test for adler32 (adler32-zlib) [ 7741.098810] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7741.390180] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 7743.072173] Key type lgssc registered [ 7744.462075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7777.972660] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7523 sec ========== 14:22:24 (1787854944) [ 7779.679912] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:22:26 (1787854946) === [ 7783.211419] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:22:30 (1787854950) === [ 7827.989286] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 7830.993177] Key type lgssc unregistered [ 7831.333123] LNet: 243547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7831.345901] LNetError: 243547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7831.356633] LNet: Removed LNI 192.168.203.149@tcp [ 7832.407727] Key type .llcrypt unregistered [ 7832.410978] Key type ._llcrypt unregistered