[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496717112 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003228] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005023] kvm-guest: setup PV IPIs [ 0.007780] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009026] pid_max: default: 32768 minimum: 301 [ 0.010172] LSM: Security Framework initializing [ 0.011069] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013099] *** VALIDATE selinux *** [ 0.021728] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026786] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029151] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030183] *** VALIDATE tmpfs *** [ 0.032424] *** VALIDATE proc *** [ 0.033344] *** VALIDATE cgroup *** [ 0.034017] *** VALIDATE cgroup2 *** [ 0.035341] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036188] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040020] Speculative Store Bypass: Vulnerable [ 0.043737] debug: unmapping init [mem 0xffffffff8ec59000-0xffffffff8ec60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046852] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048019] ... bit width: 48 [ 0.049018] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051025] ... max period: 00007fffffffffff [ 0.052020] ... fixed-purpose events: 3 [ 0.053022] ... event mask: 000000070000000f [ 0.054413] rcu: Hierarchical SRCU implementation. [ 0.056786] smp: Bringing up secondary CPUs ... [ 0.057691] x86: Booting SMP configuration: [ 0.058044] .... node #0, CPUs: #1 #2 #3 [ 0.064393] smp: Brought up 1 node, 4 CPUs [ 0.066026] smpboot: Max logical packages: 1 [ 0.067021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.291037] node 0 deferred pages initialised in 222ms [ 0.295169] devtmpfs: initialized [ 0.296394] x86/mm: Memory block size: 128MB [ 0.299245] gcov: version magic: 0x41383552 [ 0.302342] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.305130] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.308413] pinctrl core: initialized pinctrl subsystem [ 0.310247] [ 0.310927] ************************************************************* [ 0.313026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.315015] ** ** [ 0.318021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.321020] ** ** [ 0.323014] ** This means that this kernel is built to expose internal ** [ 0.326021] ** IOMMU data structures, which may compromise security on ** [ 0.328017] ** your system. ** [ 0.331018] ** ** [ 0.333017] ** If you see this message and you are not debugging the ** [ 0.335020] ** kernel, report this immediately to your vendor! ** [ 0.338018] ** ** [ 0.340018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.343024] ************************************************************* [ 0.345825] NET: Registered protocol family 16 [ 0.348497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.353090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.356098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.359161] cpuidle: using governor menu [ 0.360796] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.363510] PCI: Using configuration type 1 for base access [ 0.366162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.375083] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.378049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.382000] cryptd: max_cpu_qlen set to 1000 [ 0.384314] ACPI: Added _OSI(Module Device) [ 0.386045] ACPI: Added _OSI(Processor Device) [ 0.388028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.390020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.394141] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.400444] ACPI: Interpreter enabled [ 0.402079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.403012] ACPI: Using IOAPIC for interrupt routing [ 0.404118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.406362] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.413498] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.416060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.418024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.422098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.427000] acpiphp: Slot [2] registered [ 0.429231] acpiphp: Slot [5] registered [ 0.431117] acpiphp: Slot [6] registered [ 0.432307] acpiphp: Slot [7] registered [ 0.434191] acpiphp: Slot [8] registered [ 0.436226] acpiphp: Slot [9] registered [ 0.437206] acpiphp: Slot [10] registered [ 0.439191] acpiphp: Slot [3] registered [ 0.440153] acpiphp: Slot [4] registered [ 0.442140] acpiphp: Slot [11] registered [ 0.443133] acpiphp: Slot [12] registered [ 0.445187] acpiphp: Slot [13] registered [ 0.446156] acpiphp: Slot [14] registered [ 0.448141] acpiphp: Slot [15] registered [ 0.449132] acpiphp: Slot [16] registered [ 0.451147] acpiphp: Slot [17] registered [ 0.452111] acpiphp: Slot [18] registered [ 0.454123] acpiphp: Slot [19] registered [ 0.455163] acpiphp: Slot [20] registered [ 0.456128] acpiphp: Slot [21] registered [ 0.458123] acpiphp: Slot [22] registered [ 0.460078] acpiphp: Slot [23] registered [ 0.461175] acpiphp: Slot [24] registered [ 0.462129] acpiphp: Slot [25] registered [ 0.464126] acpiphp: Slot [26] registered [ 0.465194] acpiphp: Slot [27] registered [ 0.467149] acpiphp: Slot [28] registered [ 0.469143] acpiphp: Slot [29] registered [ 0.470172] acpiphp: Slot [30] registered [ 0.472128] acpiphp: Slot [31] registered [ 0.473077] PCI host bridge to bus 0000:00 [ 0.474030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.477034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.479047] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.481038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.484040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.487070] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.490222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.493329] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.497682] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.509024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.514047] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.517025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.519048] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.522024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.524616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.527652] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.529046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.533081] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.538019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.551738] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.555018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.561430] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.568023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.576024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.593021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.609517] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.617025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.623025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.642027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.653200] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.662026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.669025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.690027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.707965] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.719027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.730025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.748020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.759485] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.780023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.801029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.836019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.848478] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.871021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.889020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.923023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.936426] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.939358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.942356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.944485] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.947227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.954166] iommu: Default domain type: Passthrough [ 0.956484] SCSI subsystem initialized [ 0.958150] ACPI: bus type USB registered [ 0.960121] usbcore: registered new interface driver usbfs [ 0.962120] usbcore: registered new interface driver hub [ 0.964087] usbcore: registered new device driver usb [ 0.966163] pps_core: LinuxPPS API ver. 1 registered [ 0.967012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.971089] PTP clock support registered [ 0.974050] EDAC MC: Ver: 3.0.0 [ 0.976260] PCI: Using ACPI for IRQ routing [ 0.978868] NetLabel: Initializing [ 0.980012] NetLabel: domain hash size = 128 [ 0.982020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.984084] NetLabel: unlabeled traffic allowed by default [ 0.987154] vgaarb: loaded [ 0.988356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.990021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.996000] clocksource: Switched to clocksource kvm-clock [ 1.107853] VFS: Disk quotas dquot_6.6.0 [ 1.109368] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.111854] *** VALIDATE ramfs *** [ 1.113192] *** VALIDATE hugetlbfs *** [ 1.114644] pnp: PnP ACPI init [ 1.116855] pnp: PnP ACPI: found 6 devices [ 1.134376] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.139044] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.142310] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.145244] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.148694] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.152209] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.156357] NET: Registered protocol family 2 [ 1.159796] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.166279] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.171707] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.179215] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.185435] TCP: Hash tables configured (established 65536 bind 65536) [ 1.189786] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.191869] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.194818] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.197861] NET: Registered protocol family 1 [ 1.200488] RPC: Registered named UNIX socket transport module. [ 1.202841] RPC: Registered udp transport module. [ 1.204783] RPC: Registered tcp transport module. [ 1.206767] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.209445] NET: Registered protocol family 44 [ 1.211266] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.213677] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.216156] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.218376] PCI: CLS 0 bytes, default 64 [ 1.220704] Unpacking initramfs... [ 2.591755] debug: unmapping init [mem 0xffff9ac57cc54000-0xffff9ac57ffbffff] [ 2.594917] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.597410] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.600490] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.118710] Initialise system trusted keyrings [ 3.120372] Key type blacklist registered [ 3.122185] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.130256] zbud: loaded [ 3.132981] *** VALIDATE nfs *** [ 3.133886] *** VALIDATE nfs4 *** [ 3.135235] pstore: using deflate compression [ 3.138384] Platform Keyring initialized [ 3.239606] NET: Registered protocol family 38 [ 3.240878] Key type asymmetric registered [ 3.241935] Asymmetric key parser 'x509' registered [ 3.243288] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.246236] io scheduler mq-deadline registered [ 3.248114] io scheduler kyber registered [ 3.250034] io scheduler bfq registered [ 3.252414] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.255728] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.258767] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.262081] ACPI: Power Button [PWRF] [ 3.267718] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.274333] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.367678] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.404233] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.433046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.461077] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.497398] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.501610] Non-volatile memory driver v1.3 [ 3.503195] Linux agpgart interface v0.103 [ 3.534857] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.537865] vda: detected capacity change from 0 to 74760192 [ 3.552878] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.555850] vdb: detected capacity change from 0 to 1073741824 [ 3.570376] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.572631] vdc: detected capacity change from 0 to 2621440000 [ 3.609486] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.612848] vdd: detected capacity change from 0 to 2621440000 [ 3.630410] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.632789] vde: detected capacity change from 0 to 4294967296 [ 3.652363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.656762] vdf: detected capacity change from 0 to 4294967296 [ 3.679265] libphy: Fixed MDIO Bus: probed [ 3.698536] usbcore: registered new interface driver usbserial_generic [ 3.702528] usbserial: USB Serial support registered for generic [ 3.706164] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.712514] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.715366] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.718682] mousedev: PS/2 mouse device common for all mice [ 3.723279] rtc_cmos 00:05: RTC can wake from S4 [ 3.728902] rtc_cmos 00:05: registered as rtc0 [ 3.732436] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.734235] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.737803] intel_pstate: CPU model not supported [ 3.742839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.745848] hid: raw HID events driver (C) Jiri Kosina [ 3.750390] usbcore: registered new interface driver usbhid [ 3.750457] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.752499] usbhid: USB HID core driver [ 3.756608] drop_monitor: Initializing network drop monitor service [ 3.758804] Initializing XFRM netlink socket [ 3.760451] NET: Registered protocol family 10 [ 3.762851] Segment Routing with IPv6 [ 3.764157] NET: Registered protocol family 17 [ 3.765776] mpls_gso: MPLS GSO support [ 3.770199] RAS: Correctable Errors collector initialized. [ 3.772045] AVX version of gcm_enc/dec engaged. [ 3.773403] AES CTR mode by8 optimization enabled [ 3.854807] sched_clock: Marking stable (3854777031, 0)->(4894486281, -1039709250) [ 3.859465] registered taskstats version 1 [ 3.861873] Loading compiled-in X.509 certificates [ 3.864993] zswap: loaded using pool lzo/zbud [ 3.889804] Key type big_key registered [ 3.902657] Key type encrypted registered [ 3.904431] ima: No TPM chip found, activating TPM-bypass! [ 3.906145] ima: Allocated hash algorithm: sha1 [ 3.907848] ima: No architecture policies found [ 3.909541] evm: Initialising EVM extended attributes: [ 3.911304] evm: security.selinux [ 3.912310] evm: security.ima [ 3.913195] evm: security.capability [ 3.914416] evm: HMAC attrs: 0x1 [ 3.916639] rtc_cmos 00:05: setting system clock to 2026-08-31 04:39:23 UTC (1788151163) [ 3.922675] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 3.925523] debug: unmapping init [mem 0xffffffff8e982000-0xffffffff8ec58fff] [ 3.934120] Write protecting the kernel read-only data: 28672k [ 3.937087] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 3.939569] debug: unmapping init [mem 0xffffffff8d914000-0xffffffff8d9fffff] [ 3.973109] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.979195] systemd[1]: Detected virtualization kvm. [ 3.980968] systemd[1]: Detected architecture x86-64. [ 3.982681] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.008041] systemd[1]: No hostname configured. [ 4.009833] systemd[1]: Set hostname to . [ 4.011883] random: systemd: uninitialized urandom read (16 bytes read) [ 4.014328] systemd[1]: Initializing machine ID from random generator. [ 4.074159] random: ln: uninitialized urandom read (6 bytes read) [ 4.154541] random: systemd: uninitialized urandom read (16 bytes read) [ 4.156990] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.160376] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.164515] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.828276] device-mapper: uevent: version 1.0.3 [ 4.830273] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.499315] random: fast init done [ 5.513343] virtio_net virtio0 ens2: renamed from eth0 [ 5.572929] scsi host0: ata_piix [ 5.578471] scsi host1: ata_piix [ 5.580674] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.584415] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.304585] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.428803] random: crng init done [ 10.430358] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.840786] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.044607] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.313124] SELinux: Disabled at runtime. [ 12.367335] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.376313] systemd[1]: Detected virtualization kvm. [ 12.378377] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.861229] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.864624] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.869583] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.875482] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.879081] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.887453] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.898462] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ 12.970270] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.433367] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.750466] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.804146] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.946802] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.966203] EDAC sbridge: Ver: 1.1.2 [ 15.346948] Key type dns_resolver registered [ 15.680855] NFS: Registering the id_resolver key type [ 15.682659] Key type id_resolver registered [ 15.684015] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 41.717330] libcfs: loading out-of-tree module taints kernel. [ 41.737288] Key type ._llcrypt registered [ 41.738457] Key type .llcrypt registered [ 41.789280] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 58.230082] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 60.311351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.338909] alg: No test for adler32 (adler32-zlib) [ 62.008939] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 63.067343] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 64.839224] Key type lgssc registered [ 66.629176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.043010] hrtimer: interrupt took 1972691 ns [ 87.306337] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.376243] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 143.635443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 143.664368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.864660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.885518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.948393] Lustre: lustre-MDT0000: new disk, initializing [ 145.013919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.027872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 149.312710] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.870040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.137857] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.251098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.264842] Lustre: Skipped 1 previous similar message [ 164.434604] Lustre: lustre-MDT0001: new disk, initializing [ 164.539031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.609228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.632712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.094701] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.888154] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.707771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.947950] Lustre: lustre-OST0000: new disk, initializing [ 186.954287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.960635] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.030810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.553264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.563768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.632547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.594940] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.817639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.034925] Lustre: lustre-OST0001: new disk, initializing [ 209.042162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.044593] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.112486] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.585444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 215.599168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 215.635812] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.682169] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 227.684722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.285990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.912501] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.862568] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 256.438285] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 259.632549] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 263.159220] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 265.102402] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:43:42 EDT 2026 [ 287.273940] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 289.174061] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 307.686792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.689054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.705484] Lustre: Skipped 1 previous similar message [ 307.720532] Lustre: Skipped 3 previous similar messages [ 312.045972] Lustre: server umount lustre-MDT0000 complete [ 315.734510] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788151475 with bad export cookie 14358748828615182787 [ 315.739185] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.744827] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 317.920217] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.922070] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.937244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.938637] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 317.972028] Lustre: Skipped 3 previous similar messages [ 320.992502] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.994459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.018192] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 321.049812] Lustre: Skipped 2 previous similar messages [ 326.111989] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.112033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.139566] LustreError: 6512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 326.153853] Lustre: Skipped 1 previous similar message [ 330.208170] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.521095] Lustre: server umount lustre-MDT0001 complete [ 334.758570] Lustre: server umount lustre-OST0000 complete [ 338.931793] Lustre: server umount lustre-OST0001 complete [ 348.661310] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 362.067861] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 410.623106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.844833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.869378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.929700] Lustre: lustre-MDT0000: new disk, initializing [ 410.977879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.986236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.116483] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 428.218665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.400199] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 428.487931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 428.494890] Lustre: Skipped 1 previous similar message [ 428.638685] Lustre: lustre-MDT0001: new disk, initializing [ 428.771457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 428.861862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 428.881601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 433.405497] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 444.083579] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 451.336619] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 457.717837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.933173] Lustre: lustre-OST0000: new disk, initializing [ 457.936041] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.939455] Lustre: 17045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 458.038959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.187679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 459.204309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.334100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 464.345524] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 474.986896] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.671988] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 480.915636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 485.666557] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 485.816765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 489.966063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.975804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.986802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.990668] Lustre: Skipped 1 previous similar message [ 500.207072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.213848] Lustre: Skipped 4 previous similar messages [ 502.241440] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 502.526952] Lustre: server umount lustre-OST0000 complete [ 510.436073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.136310] Lustre: server umount lustre-MDT0000 complete [ 515.553623] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.575344] LustreError: 15086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 517.315204] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788151676 with bad export cookie 14358748828615184320 [ 517.316621] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.340262] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 517.668803] Lustre: server umount lustre-MDT0001 complete [ 521.625422] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:47:59 (1788151679) [ 529.590298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.872474] LustreError: 19310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.931446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.801612] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 540.130208] LustreError: 19311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.144196] LustreError: 19311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 544.398294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.368361] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 559.284744] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 568.268932] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.572926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.191303] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 592.306345] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 615.393664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.409208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 615.423948] Lustre: Skipped 1 previous similar message [ 615.425063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 615.425068] Lustre: Skipped 2 previous similar messages [ 618.643830] Lustre: server umount lustre-OST0000 complete [ 625.633511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.657899] Lustre: Skipped 2 previous similar messages [ 629.002871] Lustre: server umount lustre-MDT0000 complete [ 630.755571] LustreError: 19311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.779094] LustreError: 19311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 633.323986] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788151792 with bad export cookie 14358748828615185195 [ 633.333668] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.340811] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 633.658630] Lustre: server umount lustre-MDT0001 complete [ 646.923882] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 650.338895] Key type lgssc unregistered [ 650.707726] LNet: 22870:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.715534] LNetError: 22870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.727156] LNet: Removed LNI 192.168.202.136@tcp [ 651.932330] Key type .llcrypt unregistered [ 651.936542] Key type ._llcrypt unregistered [ 680.057184] Key type ._llcrypt registered [ 680.059507] Key type .llcrypt registered [ 680.146609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 680.854076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.967086] alg: No test for adler32 (adler32-zlib) [ 682.569098] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 682.883992] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 684.623661] Key type lgssc registered [ 686.001517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.201090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.226289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.842807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.653139] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 708.064262] LustreError: 24283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 708.088929] LustreError: 24283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 712.510697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.991564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 717.493935] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 724.731880] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 731.030058] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 739.729997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 740.121751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.281492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 748.435763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 757.411354] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 779.237063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.238725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.263438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 782.480596] Lustre: server umount lustre-OST0000 complete [ 789.481096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.486637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.500190] Lustre: Skipped 2 previous similar messages [ 789.514501] Lustre: Skipped 2 previous similar messages [ 793.240765] Lustre: server umount lustre-MDT0000 complete [ 794.596949] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 794.623435] LustreError: 24284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 797.514348] LustreError: 26128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788151957 with bad export cookie 4575790698616614144 [ 797.520522] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 797.529162] LustreError: 26128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 798.033518] Lustre: server umount lustre-MDT0001 complete [ 811.681143] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 814.619830] Key type lgssc unregistered [ 815.031460] LNet: 28263:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.042735] LNetError: 28263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.061805] LNet: Removed LNI 192.168.202.136@tcp [ 816.037298] Key type .llcrypt unregistered [ 816.039945] Key type ._llcrypt unregistered [ 844.002498] Key type ._llcrypt registered [ 844.005258] Key type .llcrypt registered [ 844.134038] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 844.995894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.039653] alg: No test for adler32 (adler32-zlib) [ 846.137650] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 846.345899] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 847.991301] Key type lgssc registered [ 849.172893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.629838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.640741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.259840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.246693] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 867.809358] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 867.850254] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 872.933227] LustreError: 29677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.363748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.899654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 879.429347] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 887.305091] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.446573] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.178280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.499326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 914.409405] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 922.861923] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.990848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 938.467398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.468847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 938.489195] Lustre: Skipped 1 previous similar message [ 938.505310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.756364] Lustre: server umount lustre-OST0000 complete [ 948.706811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.725621] Lustre: Skipped 1 previous similar message [ 948.737175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.745816] Lustre: Skipped 2 previous similar messages [ 953.367306] Lustre: server umount lustre-MDT0000 complete [ 953.824688] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.842704] LustreError: 29676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 957.737553] LustreError: 30411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152117 with bad export cookie 538571971869623226 [ 957.744531] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 957.758642] LustreError: 30411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 958.394767] Lustre: server umount lustre-MDT0001 complete [ 970.908392] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 974.055929] Key type lgssc unregistered [ 974.430809] LNet: 33582:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.450792] LNetError: 33582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.530371] LNet: Removed LNI 192.168.202.136@tcp [ 976.510752] Key type .llcrypt unregistered [ 976.516682] Key type ._llcrypt unregistered [ 1012.393456] Key type ._llcrypt registered [ 1012.396852] Key type .llcrypt registered [ 1012.498144] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1013.883643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.920068] alg: No test for adler32 (adler32-zlib) [ 1014.941614] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1015.146126] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1016.791242] Key type lgssc registered [ 1017.849757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.548285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1026.570316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.981994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.253778] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1038.304832] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.326766] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1042.918897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.419787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.774129] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1057.699502] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.658975] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.613322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.054868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.095851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1079.430139] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1086.755181] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1098.729439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.733400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.748140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.763681] Lustre: Skipped 1 previous similar message [ 1098.796281] Lustre: Skipped 1 previous similar message [ 1101.525440] Lustre: server umount lustre-OST0000 complete [ 1108.964934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.974380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.988047] Lustre: Skipped 1 previous similar message [ 1109.010511] Lustre: Skipped 1 previous similar message [ 1112.607355] Lustre: server umount lustre-MDT0000 complete [ 1114.100486] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.136885] LustreError: 35000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1118.285330] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152277 with bad export cookie 6718737912611748127 [ 1118.286378] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.290282] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1118.610810] Lustre: server umount lustre-MDT0001 complete [ 1134.824399] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1137.404672] Key type lgssc unregistered [ 1137.647745] LNet: 38556:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.654214] LNetError: 38556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.668852] LNet: Removed LNI 192.168.202.136@tcp [ 1138.810634] Key type .llcrypt unregistered [ 1138.812891] Key type ._llcrypt unregistered [ 1165.964303] Key type ._llcrypt registered [ 1165.969945] Key type .llcrypt registered [ 1166.126802] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1167.423572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.471533] alg: No test for adler32 (adler32-zlib) [ 1168.526777] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1168.859301] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1170.511175] Key type lgssc registered [ 1171.503985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1178.369909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1178.384386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.733579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.207478] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1189.875615] LustreError: 39969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.910567] LustreError: 39969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1192.973266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.228492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.249068] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1207.056534] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.735962] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.304499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1221.798124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1224.873128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1228.304675] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1236.155700] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.669748] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1248.742490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.754566] Lustre: Skipped 1 previous similar message [ 1248.757639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1253.784174] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1253.790952] Lustre: Skipped 2 previous similar messages [ 1258.876964] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1258.886503] Lustre: Skipped 2 previous similar messages [ 1259.999327] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1260.164217] Lustre: server umount lustre-OST0000 complete [ 1284.576082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.577272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.591554] Lustre: Skipped 1 previous similar message [ 1284.608799] Lustre: Skipped 3 previous similar messages [ 1288.738859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1288.753367] Lustre: Skipped 1 previous similar message [ 1295.329265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1295.481592] Lustre: server umount lustre-MDT0000 complete [ 1298.914715] LustreError: 41616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.944487] LustreError: 41616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1299.025487] LustreError: 42673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152458 with bad export cookie 3256536121120111786 [ 1299.029265] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1299.039421] LustreError: 42673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1305.459783] Lustre: server umount lustre-MDT0001 complete [ 1318.296884] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1320.813875] Key type lgssc unregistered [ 1321.073675] LNet: 43692:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.084187] LNetError: 43692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.114075] LNet: Removed LNI 192.168.202.136@tcp [ 1321.849086] Key type .llcrypt unregistered [ 1321.852823] Key type ._llcrypt unregistered [ 1347.294125] Key type ._llcrypt registered [ 1347.297139] Key type .llcrypt registered [ 1347.391397] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1348.766080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1348.832093] alg: No test for adler32 (adler32-zlib) [ 1349.919834] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1350.189831] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1351.855174] Key type lgssc registered [ 1352.910442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1360.685574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1360.704652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.320772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.713864] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1371.628076] LustreError: 45115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.649526] LustreError: 45115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1376.616648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.739587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1376.930564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.441092] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1390.392561] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1396.349518] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.114065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.453292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.543181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1410.219474] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1418.769737] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1428.451924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.469332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.494858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1430.933115] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1432.544330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.561089] Lustre: Skipped 1 previous similar message [ 1434.405403] Lustre: server umount lustre-MDT0000 complete [ 1436.018400] LustreError: 46093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.664317] LustreError: 45115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.673000] LustreError: 45115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1438.299535] LustreError: 45096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152597 with bad export cookie 11590809198329535598 [ 1438.311115] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.322509] LustreError: 45096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1441.136633] LustreError: 45109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.196229] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [ 1441.202845] Lustre: Skipped 3 previous similar messages [ 1442.794382] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.781566] Lustre: server umount lustre-MDT0001 complete [ 1459.909380] Lustre: server umount lustre-OST0000 complete [ 1477.034400] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1480.065201] Key type lgssc unregistered [ 1480.408552] LNet: 48974:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.418423] LNetError: 48974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.445303] LNet: Removed LNI 192.168.202.136@tcp [ 1482.291955] Key type .llcrypt unregistered [ 1482.294047] Key type ._llcrypt unregistered [ 1505.525592] Key type ._llcrypt registered [ 1505.529874] Key type .llcrypt registered [ 1505.639734] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1506.206531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.213504] alg: No test for adler32 (adler32-zlib) [ 1507.269541] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1507.461981] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1509.127388] Key type lgssc registered [ 1510.062148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1516.718391] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1516.734833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.260421] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1537.376561] LustreError: 50339:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 1563.999307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.968223] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1577.286606] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.476168] Lustre: server umount lustre-OST0000 complete [ 1760.915382] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1763.500898] Key type lgssc unregistered [ 1763.809790] LNet: 52013:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.815832] LNetError: 52013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.829934] LNet: Removed LNI 192.168.202.136@tcp [ 1764.765775] Key type .llcrypt unregistered [ 1764.767414] Key type ._llcrypt unregistered [ 1789.488753] Key type ._llcrypt registered [ 1789.494417] Key type .llcrypt registered [ 1789.625326] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1790.439408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1790.481433] alg: No test for adler32 (adler32-zlib) [ 1791.622610] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1792.002314] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1793.711562] Key type lgssc registered [ 1794.930229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1802.298950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1802.319176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.647458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.357622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1813.986589] LustreError: 53428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.026594] LustreError: 53428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1817.273736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.643921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1821.813726] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1830.109702] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1835.871386] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1842.372321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.610642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.886716] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1848.770381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1855.140498] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.040177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.042159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.054241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.057450] Lustre: Skipped 1 previous similar message [ 1859.066638] Lustre: Skipped 1 previous similar message [ 1864.162915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.180065] Lustre: Skipped 1 previous similar message [ 1869.282247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.288520] Lustre: Skipped 1 previous similar message [ 1871.327312] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1871.497995] Lustre: server umount lustre-OST0000 complete [ 1879.026557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1879.037673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.056304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.644279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.647178] Lustre: Skipped 2 previous similar messages [ 1889.247755] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1889.380938] Lustre: server umount lustre-MDT0000 complete [ 1889.766756] LustreError: 53425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.790064] LustreError: 53425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1892.775671] LustreError: 53409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153052 with bad export cookie 10377464273741894498 [ 1892.778901] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1892.786870] LustreError: 53409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1893.168329] Lustre: server umount lustre-MDT0001 complete [ 1905.560268] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1907.975392] Key type lgssc unregistered [ 1908.263956] LNet: 56960:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.267333] LNetError: 56960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.295809] LNet: Removed LNI 192.168.202.136@tcp [ 1909.283360] Key type .llcrypt unregistered [ 1909.285650] Key type ._llcrypt unregistered [ 1933.960751] Key type ._llcrypt registered [ 1933.963648] Key type .llcrypt registered [ 1934.056691] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1935.068536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.387884] alg: No test for adler32 (adler32-zlib) [ 1936.392857] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1936.562701] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1938.207241] Key type lgssc registered [ 1939.122950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1947.164377] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.191376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.783498] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1967.903416] LustreError: 58351:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 1994.591405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.214548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2008.440444] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2016.234348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.355973] LustreError: 57686:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006c460f88 x1875014420793600/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2016.756832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2016.832667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2021.159996] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2021.864768] LustreError: 58376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.879929] LustreError: 58376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2026.990199] LustreError: 58377:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.047814] LustreError: 59387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2029.073732] LustreError: 59387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2030.173040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.594871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2035.601669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2044.043786] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.692760] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.161718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2056.171389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.177875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.206945] Lustre: Skipped 1 previous similar message [ 2058.922226] Lustre: server umount lustre-OST0000 complete [ 2080.736995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.753466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.759455] Lustre: Skipped 1 previous similar message [ 2080.773442] Lustre: Skipped 2 previous similar messages [ 2085.856393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.865920] Lustre: Skipped 1 previous similar message [ 2086.780302] Lustre: server umount lustre-MDT0000 complete [ 2090.544450] LustreError: 58357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153250 with bad export cookie 14787267493251270231 [ 2090.557783] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2090.559341] LustreError: 58357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2097.079205] Lustre: server umount lustre-MDT0001 complete [ 2110.508296] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2114.415781] Key type lgssc unregistered [ 2114.854986] LNet: 62100:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.869452] LNetError: 62100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.904344] LNet: Removed LNI 192.168.202.136@tcp [ 2116.307267] Key type .llcrypt unregistered [ 2116.309068] Key type ._llcrypt unregistered [ 2146.468435] Key type ._llcrypt registered [ 2146.471779] Key type .llcrypt registered [ 2146.728822] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2148.251750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.318741] alg: No test for adler32 (adler32-zlib) [ 2149.478866] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2149.931914] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2151.728602] Key type lgssc registered [ 2153.712581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2164.244920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2164.258943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.768264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2170.662611] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2175.975439] LustreError: 63513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2176.019659] LustreError: 63513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2179.711793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.949063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2184.699691] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2193.412303] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2200.516504] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2207.908357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.361977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2211.455164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2216.048925] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2225.132801] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.772762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.777881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.799765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2253.504896] Lustre: server umount lustre-OST0000 complete [ 2260.961079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.969365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.976793] Lustre: Skipped 1 previous similar message [ 2263.375820] Lustre: server umount lustre-MDT0000 complete [ 2266.081924] LustreError: 63514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.850268] LustreError: 63496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153426 with bad export cookie 16835083204101638622 [ 2266.851432] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.858043] LustreError: 63496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2277.483374] Lustre: server umount lustre-MDT0001 complete [ 2290.673525] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2293.348784] Key type lgssc unregistered [ 2293.564509] LNet: 67072:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.569846] LNetError: 67072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.581796] LNet: Removed LNI 192.168.202.136@tcp [ 2294.311180] Key type .llcrypt unregistered [ 2294.313706] Key type ._llcrypt unregistered [ 2321.146349] Key type ._llcrypt registered [ 2321.150253] Key type .llcrypt registered [ 2321.365287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2322.652741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.669300] alg: No test for adler32 (adler32-zlib) [ 2323.853778] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2324.166391] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2325.911448] Key type lgssc registered [ 2327.087923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2336.756641] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:18:14 (1788153494) [ 2343.062766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.084579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.488396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.848952] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2354.660577] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.670379] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2359.101919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.336254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2363.770390] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2371.503517] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2377.666458] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2385.493186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.934216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2390.075809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2394.042047] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2403.447948] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2409.670656] Lustre: Failing over lustre-MDT0000 [ 2409.971366] Lustre: server umount lustre-MDT0000 complete [ 2410.398068] LustreError: 68772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.471032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2410.479914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.577911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.582291] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.603067] Lustre: Skipped 1 previous similar message [ 2414.619122] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2419.683980] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.715900] LustreError: 68778:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2419.903152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.987278] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.069355] Lustre: *** cfs_fail_loc=135, val=0*** [ 2420.137628] LustreError: 71866:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2420.143456] LustreError: 71866:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -2 [ 2420.151901] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2420.151901] [ 2420.168149] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2420.177075] LustreError: 71854:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2420.183972] LustreError: 71854:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 2420.195979] LustreError: 71854:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2420.201274] LustreError: 71854:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2420.224037] Lustre: server umount lustre-MDT0000 complete [ 2420.227326] LustreError: 71854:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2424.807927] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.829246] LustreError: 69505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2427.283396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2427.401161] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2427.653287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.738179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2430.831195] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.630241] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2433.017688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2433.064385] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2433.104310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2440.160670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2440.165682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.192870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.186362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.203333] Lustre: Skipped 1 previous similar message [ 2445.227471] Lustre: server umount lustre-OST0000 complete [ 2451.425077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.447927] Lustre: Skipped 1 previous similar message [ 2451.452980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.646296] Lustre: server umount lustre-MDT0000 complete [ 2456.553852] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.589022] LustreError: 68777:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2460.281334] LustreError: 68759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153619 with bad export cookie 12407399282005508708 [ 2460.285350] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.295746] LustreError: 68759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2460.937744] Lustre: server umount lustre-MDT0001 complete [ 2471.306504] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2474.145904] Key type lgssc unregistered [ 2474.505720] LNet: 73646:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.520828] LNetError: 73646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.535628] LNet: Removed LNI 192.168.202.136@tcp [ 2475.307921] Key type .llcrypt unregistered [ 2475.311852] Key type ._llcrypt unregistered [ 2499.568925] Key type ._llcrypt registered [ 2499.571378] Key type .llcrypt registered [ 2499.682258] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2501.039937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2501.065064] alg: No test for adler32 (adler32-zlib) [ 2502.137651] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2502.381110] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2504.055502] Key type lgssc registered [ 2504.976412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.802844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.823642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.210110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.678509] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2522.596325] LustreError: 75058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.606765] LustreError: 75058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2525.281604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2525.748597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2529.379202] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2537.499561] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2543.253306] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2551.217808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.558502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.682837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2558.039332] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2567.343985] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2576.357540] Lustre: Failing over lustre-MDT0000 [ 2576.362615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.390504] Lustre: Skipped 1 previous similar message [ 2576.396691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.404678] Lustre: Skipped 2 previous similar messages [ 2576.740853] Lustre: server umount lustre-MDT0000 complete [ 2579.347276] LustreError: 75054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.476383] LustreError: 75059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2581.504812] LustreError: 75059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2584.443494] LustreError: 75055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2587.542906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.673237] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.816610] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2587.921583] LustreError: 78146:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2587.926173] LustreError: 78146:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -9 [ 2587.931736] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2587.931736] [ 2587.936423] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2587.946070] LustreError: 78134:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2587.950237] LustreError: 78134:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -9 [ 2587.955666] LustreError: 78134:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2587.959585] LustreError: 78134:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2587.977088] Lustre: server umount lustre-MDT0000 complete [ 2587.979674] LustreError: 78134:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2589.568082] LustreError: 75054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.590018] LustreError: 75054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2594.008477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.178765] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.507446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.554679] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2594.673833] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2598.556122] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2599.911517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2599.955729] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2600.036427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2607.080944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2607.086472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.101255] Lustre: Skipped 1 previous similar message [ 2607.112579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.156412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2610.164134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.288183] Lustre: server umount lustre-OST0000 complete [ 2620.384456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2620.389057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.401091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.420560] Lustre: Skipped 1 previous similar message [ 2620.434413] Lustre: Skipped 2 previous similar messages [ 2623.348839] Lustre: server umount lustre-MDT0000 complete [ 2625.504673] LustreError: 76111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.517586] LustreError: 76111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2626.485533] LustreError: 75038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153786 with bad export cookie 16184960341496241081 [ 2626.495632] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2626.495789] LustreError: 75038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2626.753295] Lustre: server umount lustre-MDT0001 complete [ 2637.839914] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2640.813852] Key type lgssc unregistered [ 2641.098587] LNet: 79926:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2641.101886] LNetError: 79926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2641.118195] LNet: Removed LNI 192.168.202.136@tcp [ 2641.879211] Key type .llcrypt unregistered [ 2641.884556] Key type ._llcrypt unregistered [ 2652.808535] Key type ._llcrypt registered [ 2652.813176] Key type .llcrypt registered [ 2652.937439] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2653.210708] Key type .llcrypt unregistered [ 2653.212693] Key type ._llcrypt unregistered [ 2676.371433] Key type ._llcrypt registered [ 2676.375525] Key type .llcrypt registered [ 2676.482449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2677.360936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2677.382734] alg: No test for adler32 (adler32-zlib) [ 2678.532267] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2678.725386] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2680.399451] Key type lgssc registered [ 2681.586454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2688.164612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.173513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.585773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.325817] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2699.747173] LustreError: 81829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.777360] LustreError: 81829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2700.959027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.268291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.478445] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2712.511703] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2718.277102] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2725.718390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.867037] LustreError: 83652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.189473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.279043] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2734.615718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2740.180221] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2751.970098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2751.992000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.013374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.044567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.047339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.047647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.072132] Lustre: Skipped 1 previous similar message [ 2758.139896] Lustre: server umount lustre-OST0000 complete [ 2762.721373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2762.729497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.743165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.283433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.284460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.700499] Lustre: server umount lustre-MDT0000 complete [ 2770.401611] LustreError: 81830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2770.437771] LustreError: 81830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2772.481528] LustreError: 81811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153932 with bad export cookie 2820603706869778638 [ 2772.485342] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.488451] LustreError: 81811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2772.728946] Lustre: server umount lustre-MDT0001 complete [ 2784.216611] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2787.054212] Key type lgssc unregistered [ 2787.312492] LNet: 85364:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.330441] LNetError: 85364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.344870] LNet: Removed LNI 192.168.202.136@tcp [ 2788.192298] Key type .llcrypt unregistered [ 2788.195524] Key type ._llcrypt unregistered [ 2812.049955] Key type ._llcrypt registered [ 2812.052760] Key type .llcrypt registered [ 2812.178436] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2813.781255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2813.832175] alg: No test for adler32 (adler32-zlib) [ 2814.829876] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2815.022995] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2816.679977] Key type lgssc registered [ 2817.686287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2825.256480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2825.269116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.966414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.800582] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2836.461162] LustreError: 86776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.480324] LustreError: 86776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2840.655686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.170956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2845.573503] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2852.945850] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2859.210657] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2866.646609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.147129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.258509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2873.875804] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2881.571383] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2892.771788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2892.779574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.800773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.857955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.860169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.869222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.888687] Lustre: Skipped 1 previous similar message [ 2897.994798] Lustre: server umount lustre-OST0000 complete [ 2902.509663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2902.517459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.538166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.089805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.089996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2906.127101] Lustre: Skipped 1 previous similar message [ 2908.622686] Lustre: server umount lustre-MDT0000 complete [ 2911.202844] LustreError: 86776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.221377] LustreError: 86776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2912.828555] LustreError: 86757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154072 with bad export cookie 7767526253864312284 [ 2912.836772] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.840356] LustreError: 86757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2913.368835] Lustre: server umount lustre-MDT0001 complete [ 2926.794417] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2930.021581] Key type lgssc unregistered [ 2930.367954] LNet: 90309:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2930.377463] LNetError: 90309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2930.414587] LNet: Removed LNI 192.168.202.136@tcp [ 2931.420242] Key type .llcrypt unregistered [ 2931.426666] Key type ._llcrypt unregistered [ 2959.252641] Key type ._llcrypt registered [ 2959.256856] Key type .llcrypt registered [ 2959.399622] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2961.096246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2961.141502] alg: No test for adler32 (adler32-zlib) [ 2962.145714] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2962.339405] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2964.047941] Key type lgssc registered [ 2965.305937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.376477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.390894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.869024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.962135] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2985.440566] LustreError: 91726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.460797] LustreError: 91726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2990.560077] LustreError: 91725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.353901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.133787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.200360] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3009.721847] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3015.909726] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3023.686920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.191107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.841589] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3039.267686] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3045.751897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3063.264567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3063.272054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3063.287134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3066.336719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3066.343036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3066.353986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.379401] Lustre: Skipped 1 previous similar message [ 3071.461652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3071.470125] Lustre: Skipped 1 previous similar message [ 3076.063134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3076.282501] Lustre: server umount lustre-OST0000 complete [ 3081.697907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3081.703162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.732244] Lustre: Skipped 1 previous similar message [ 3081.747342] Lustre: Skipped 1 previous similar message [ 3086.786491] Lustre: server umount lustre-MDT0000 complete [ 3086.816879] LustreError: 91719:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.830851] LustreError: 91719:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3091.495959] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154251 with bad export cookie 5090017402537353876 [ 3091.507607] LustreError: 91707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3091.512357] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.917508] Lustre: server umount lustre-MDT0001 complete [ 3104.669443] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3107.894461] Key type lgssc unregistered [ 3108.236765] LNet: 95286:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3108.241700] LNetError: 95286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3108.253613] LNet: Removed LNI 192.168.202.136@tcp [ 3109.013436] Key type .llcrypt unregistered [ 3109.019664] Key type ._llcrypt unregistered [ 3135.203970] Key type ._llcrypt registered [ 3135.206601] Key type .llcrypt registered [ 3135.330920] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3136.285251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3136.471277] alg: No test for adler32 (adler32-zlib) [ 3137.701421] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3138.042682] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3139.671463] Key type lgssc registered [ 3140.565240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3147.971247] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3148.004144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3163.615726] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3168.736440] LustreError: 96670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 3195.423686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3202.791203] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3211.499955] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3218.846967] Lustre: server umount lustre-OST0000 complete [ 3229.311532] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:33:06 (1788154386) [ 3237.322127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3238.147308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.496976] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.520424] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3244.112851] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3247.595255] LustreError: 98311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.706835] LustreError: 98312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.061670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.388252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.225196] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3270.138161] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3277.179316] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3284.965279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.465871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3292.815674] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3301.389925] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3308.030116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3318.244297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.244869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.274349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.274928] Lustre: Skipped 1 previous similar message [ 3323.426151] Lustre: server umount lustre-OST0000 complete [ 3328.486255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.497533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3328.521824] Lustre: Skipped 1 previous similar message [ 3328.536167] Lustre: Skipped 2 previous similar messages [ 3333.604145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.061687] Lustre: server umount lustre-MDT0000 complete [ 3337.528788] LustreError: 98292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154497 with bad export cookie 13571014551662831494 [ 3337.539985] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3337.925374] Lustre: server umount lustre-MDT0001 complete [ 3349.826685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3352.769737] Key type lgssc unregistered [ 3353.178705] LNet: 102008:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3353.181336] LNetError: 102008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.201523] LNet: Removed LNI 192.168.202.136@tcp [ 3354.166328] Key type .llcrypt unregistered [ 3354.168386] Key type ._llcrypt unregistered [ 3403.673825] Key type ._llcrypt registered [ 3403.676904] Key type .llcrypt registered [ 3403.827146] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3405.151715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3405.275545] alg: No test for adler32 (adler32-zlib) [ 3406.374818] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3406.655849] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3408.375186] Key type lgssc registered [ 3409.400651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3417.667475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3417.697466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.135807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.095531] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3429.350759] LustreError: 104211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.395464] LustreError: 104211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3432.970180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.315249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3437.850889] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3445.720914] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3452.476904] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3458.879852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.173996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.327067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3466.060273] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3475.678423] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3485.154673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3485.169078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.197206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.226327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.229775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.026946] Lustre: server umount lustre-OST0000 complete [ 3498.464704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.480781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.489422] Lustre: Skipped 2 previous similar messages [ 3501.476832] Lustre: server umount lustre-MDT0000 complete [ 3503.584606] LustreError: 105450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.608497] LustreError: 105450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3505.297929] LustreError: 106054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154664 with bad export cookie 12786951158948412398 [ 3505.304055] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.314424] LustreError: 106054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3505.804537] Lustre: server umount lustre-MDT0001 complete [ 3518.165538] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3521.483555] Key type lgssc unregistered [ 3521.857461] LNet: 107742:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3521.863504] LNetError: 107742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3521.875057] LNet: Removed LNI 192.168.202.136@tcp [ 3522.974280] Key type .llcrypt unregistered [ 3522.976263] Key type ._llcrypt unregistered [ 3542.500366] Key type ._llcrypt registered [ 3542.504796] Key type .llcrypt registered [ 3542.603715] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3543.640426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3544.048214] alg: No test for adler32 (adler32-zlib) [ 3545.075339] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3545.332361] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3547.056145] Key type lgssc registered [ 3548.182960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3556.372556] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3556.384504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.872156] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3576.991326] LustreError: 108987:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 3603.683059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3610.116815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3618.978766] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3626.167563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.448233] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3626.453229] LustreError: 110011:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3626.456861] LustreError: 110011:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 3626.461090] LustreError: 110011:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 3626.516575] Lustre: server umount lustre-MDT0000 complete [ 3626.518784] LustreError: 110011:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3645.788389] Lustre: server umount lustre-OST0000 complete [ 3659.818415] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3674.003762] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3709.437080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3709.653203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3709.676379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3709.730442] Lustre: lustre-MDT0000: new disk, initializing [ 3709.793401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.809440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3714.488058] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3724.930797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.052791] Lustre: 112819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3725.105198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3725.112981] Lustre: Skipped 1 previous similar message [ 3725.256289] Lustre: lustre-MDT0001: new disk, initializing [ 3725.480841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3725.554497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3725.571925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3730.864120] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3740.871971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3747.718500] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3755.386516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.673822] Lustre: lustre-OST0000: new disk, initializing [ 3755.676761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3755.686611] Lustre: 114785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3755.768239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.025076] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3763.779423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3763.802410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3763.879252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3771.959275] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.884354] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3777.091432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3783.029340] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3783.294152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3789.795459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.805278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.822341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.701057] Lustre: server umount lustre-OST0000 complete [ 3796.970785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3796.983976] LustreError: Skipped 1 previous similar message [ 3796.991729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3797.015978] Lustre: Skipped 1 previous similar message [ 3797.030442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3797.033330] Lustre: Skipped 1 previous similar message [ 3800.034956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.037214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3805.155369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3805.172509] Lustre: Skipped 2 previous similar messages [ 3810.273138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3810.274116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.305376] Lustre: Skipped 1 previous similar message [ 3810.557429] Lustre: server umount lustre-MDT0000 complete [ 3815.009599] LustreError: 114795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154974 with bad export cookie 14163140013092890808 [ 3815.017032] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.445360] Lustre: server umount lustre-MDT0001 complete [ 3825.765879] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:43:02 (1788154982) [ 3830.755733] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3847.659447] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3859.613109] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3894.644628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.026561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3895.074068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3895.166417] Lustre: lustre-MDT0000: new disk, initializing [ 3895.238270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.253450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3900.077584] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3911.985548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.093442] Lustre: 119688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3912.133336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3912.138049] Lustre: Skipped 1 previous similar message [ 3912.193882] Lustre: lustre-MDT0001: new disk, initializing [ 3912.299363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3912.332298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3912.343308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3917.549665] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3928.991438] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3935.562667] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.498539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.976196] Lustre: lustre-OST0000: new disk, initializing [ 3943.981836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3943.988899] Lustre: 121659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3944.109767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3950.161709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3950.175992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3950.295853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3950.816879] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3963.012056] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3968.408971] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3968.751995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3974.612290] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3974.787249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3981.281765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.288047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3981.299641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3981.306910] LustreError: Skipped 1 previous similar message [ 3981.313411] Lustre: Skipped 1 previous similar message [ 3982.939310] Lustre: server umount lustre-OST0000 complete [ 3988.960393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.972045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.980945] Lustre: Skipped 1 previous similar message [ 3993.127581] Lustre: server umount lustre-MDT0000 complete [ 3996.756258] LustreError: 121658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155156 with bad export cookie 14163140013092891690 [ 3996.766691] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.773479] LustreError: 121658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3997.201602] Lustre: server umount lustre-MDT0001 complete [ 4005.174196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4005.592072] LustreError: 123879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4005.609693] LustreError: 123879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4005.662988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.463250] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4010.979166] LustreError: 123880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.099136] LustreError: 123879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.719676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4024.963103] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4032.269838] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.653906] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4045.902773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4052.528179] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4062.289824] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4072.240759] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4077.027296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4077.034676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.045582] Lustre: Skipped 1 previous similar message [ 4077.052798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.055717] Lustre: Skipped 3 previous similar messages [ 4078.761618] Lustre: server umount lustre-OST0000 complete [ 4089.047848] Lustre: server umount lustre-MDT0000 complete [ 4091.359958] LustreError: 125119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.392374] LustreError: 125119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4092.772338] LustreError: 123861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155252 with bad export cookie 14163140013092892572 [ 4092.781583] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4092.791865] LustreError: 123861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4103.357719] Lustre: server umount lustre-MDT0001 complete [ 4114.731379] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4117.832434] Key type lgssc unregistered [ 4118.266561] LNet: 127511:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4118.280275] LNetError: 127511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4118.329043] LNet: Removed LNI 192.168.202.136@tcp [ 4119.368855] Key type .llcrypt unregistered [ 4119.370798] Key type ._llcrypt unregistered [ 4140.124946] Key type ._llcrypt registered [ 4140.128158] Key type .llcrypt registered [ 4140.270346] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4157.220630] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4158.274340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4158.381509] alg: No test for adler32 (adler32-zlib) [ 4159.510346] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4159.748448] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4161.448565] Key type lgssc registered [ 4163.135392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4204.831152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4204.872896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.136498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4206.181411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4206.306928] Lustre: lustre-MDT0000: new disk, initializing [ 4206.411671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.444600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4212.292829] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4223.104671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.275032] Lustre: 130666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4223.343766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4223.352782] Lustre: Skipped 1 previous similar message [ 4223.488350] Lustre: lustre-MDT0001: new disk, initializing [ 4223.673110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4223.750476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4223.789480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4228.286884] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4239.347856] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4246.102813] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4253.349027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4253.554323] Lustre: lustre-OST0000: new disk, initializing [ 4253.557495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4253.562726] Lustre: 132635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4253.639243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.826248] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4261.454512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4261.469546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4261.594175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4268.866513] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4274.076529] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4274.260727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4279.660799] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4279.979915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.340717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4282.357430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.409218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.418288] Lustre: Skipped 1 previous similar message [ 4287.461384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4287.468716] Lustre: Skipped 1 previous similar message [ 4288.354756] Lustre: server umount lustre-OST0000 complete [ 4292.584156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4292.590617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.608507] Lustre: Skipped 2 previous similar messages [ 4292.619647] Lustre: Skipped 1 previous similar message [ 4297.700697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4297.704713] Lustre: Skipped 1 previous similar message [ 4298.078232] Lustre: server umount lustre-MDT0000 complete [ 4302.221512] LustreError: 130659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155461 with bad export cookie 14887505211755969407 [ 4302.223415] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4302.237091] LustreError: 130659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4302.476753] Lustre: server umount lustre-MDT0001 complete [ 4310.829796] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:51:08 (1788155468) [ 4317.253204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.944286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4322.479891] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4323.300841] LustreError: 135047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.317549] LustreError: 135047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4328.416825] LustreError: 135048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.322206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4331.723564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4335.852617] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4343.336418] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4349.192660] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4352.485836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4352.497691] LustreError: Skipped 1 previous similar message [ 4352.499871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.522928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.532832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.008920] Lustre: server umount lustre-MDT0000 complete [ 4361.696378] LustreError: 135048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4361.722775] LustreError: 135048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4361.840564] LustreError: 135027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155521 with bad export cookie 14887505211755970289 [ 4361.851807] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4361.855785] LustreError: 135027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4362.574396] Lustre: server umount lustre-MDT0001 complete [ 4373.128357] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:52:10 (1788155530) [ 4380.776595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.511916] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4401.631559] LustreError: 137461:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 4427.362102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.626668] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4441.286559] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4443.349919] Lustre: server umount lustre-OST0000 complete [ 4451.424068] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:53:29 (1788155609) [ 4457.813624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.111395] LustreError: 138906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.154805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.204869] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4463.588005] LustreError: 138907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.575682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.406898] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4483.713267] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4489.984804] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4497.593530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4497.812284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.821240] Lustre: Skipped 1 previous similar message [ 4504.106837] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4512.768654] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4526.269139] Lustre: server umount lustre-MDT0000 complete [ 4527.596636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.604055] LustreError: 138906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.607738] Lustre: Skipped 2 previous similar messages [ 4527.638242] LustreError: 138906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4529.729326] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155689 with bad export cookie 14887505211755970954 [ 4529.737355] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4529.738355] LustreError: 138886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4529.967242] Lustre: server umount lustre-MDT0001 complete [ 4544.207029] Lustre: server umount lustre-OST0000 complete [ 4556.017476] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:55:13 (1788155713) [ 4564.882712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4565.555785] LustreError: 142582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.699133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.315197] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4580.369865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.248669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4593.169794] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4599.826131] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4607.475645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.010204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4615.646412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4624.052541] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4629.600413] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4629.782812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4634.312803] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4634.622622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4636.651855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4636.670887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.693161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.703953] Lustre: Skipped 4 previous similar messages [ 4642.690576] Lustre: server umount lustre-OST0000 complete [ 4652.518873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4652.525130] LustreError: Skipped 1 previous similar message [ 4654.017105] Lustre: server umount lustre-MDT0000 complete [ 4657.632826] LustreError: 143311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.658806] LustreError: 143311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4657.724656] LustreError: 142562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155817 with bad export cookie 14887505211755972018 [ 4657.734093] LustreError: 142562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4657.737051] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4658.053077] Lustre: server umount lustre-MDT0001 complete [ 4667.352660] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 01:57:05 (1788155825) [ 4674.875309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4690.464447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4695.583577] LustreError: 146745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 4721.312607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4721.324974] Lustre: Skipped 2 previous similar messages [ 4728.246395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4737.120759] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4745.110410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.659389] LustreError: 146769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.803995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4749.824985] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4758.634402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.155551] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4771.172451] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4776.710991] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4782.345964] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4782.558500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4786.590609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4786.797777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4789.740787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4789.742154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.744294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4789.744304] Lustre: Skipped 5 previous similar messages [ 4789.770871] Lustre: Skipped 5 previous similar messages [ 4795.455960] Lustre: server umount lustre-MDT0000 complete [ 4799.409128] LustreError: 146751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155958 with bad export cookie 14887505211755973047 [ 4799.421164] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4799.431768] LustreError: 146751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4799.684243] Lustre: server umount lustre-MDT0001 complete [ 4812.802977] Lustre: server umount lustre-OST0000 complete [ 4822.751909] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 01:59:40 (1788155980) [ 4831.077267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.816514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4851.936511] LustreError: 150938:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 4884.283724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4892.636930] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4900.816658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.934450] LustreError: 128888:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e2d10d0 x1875016751928576/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4901.182569] LustreError: 150963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.200808] LustreError: 150963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4901.319085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4905.992218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4916.370031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.693299] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4931.644867] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4939.004580] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4946.702305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4947.020823] Lustre: lustre-OST0001: new disk, initializing [ 4947.023780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4947.036431] Lustre: 153823:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4955.282595] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4956.791502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4956.804506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4956.990669] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4965.313382] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4970.084603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4970.244973] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4975.416695] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4975.605484] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4977.647578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.653288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4977.669371] Lustre: Skipped 1 previous similar message [ 4977.687381] Lustre: Skipped 6 previous similar messages [ 4983.706204] Lustre: server umount lustre-OST0000 complete [ 4993.737631] Lustre: server umount lustre-OST0001 complete [ 5004.030966] Lustre: server umount lustre-MDT0000 complete [ 5008.129617] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156167 with bad export cookie 14887505211755973999 [ 5008.137041] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5008.144637] LustreError: 150944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5008.748430] Lustre: server umount lustre-MDT0001 complete [ 5035.996453] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:03:13 (1788156193) [ 5037.701344] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5039.730581] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5042.302597] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:03:19 (1788156199) [ 5049.578219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5049.960649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5050.017828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5050.309216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.317665] Lustre: Skipped 6 previous similar messages [ 5056.437332] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5068.117553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.269735] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5068.295923] Lustre: 157381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5073.620451] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5082.351510] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5088.625203] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5097.194658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.390789] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5104.593333] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5105.302128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5112.696426] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5117.835685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5118.113800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5122.287032] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5122.462292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5125.602687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5125.612744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.620573] Lustre: Skipped 4 previous similar messages [ 5125.624272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.627220] Lustre: Skipped 10 previous similar messages [ 5130.810751] Lustre: server umount lustre-OST0000 complete [ 5153.373324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5159.908642] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5166.118318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.133249] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5168.008493] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5174.958512] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5175.337647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5176.360465] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.386697] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5180.632941] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5180.884982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5187.315348] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff888c0539f800.ost_server_uuid 50 [ 5189.015293] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff888c0539f800.ost_server_uuid in FULL state after 0 sec [ 5189.382502] LustreError: 158842:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.36@tcp) returned error from blocking AST (req@00000000462db4d5 x1875016752035200 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e6c9683e/0xce9b0b1cda7dd6c9 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.36@tcp remote: 0x7df712b327a48155 expref: 5 pid: 158842 timeout: 5289 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5189.427597] LustreError: lustre-MDT0000: A client on nid 192.168.202.36@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5189.453930] LustreError: 157378:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.36@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e6c9683e/0xce9b0b1cda7dd6c9 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.36@tcp remote: 0x7df712b327a48155 expref: 6 pid: 158842 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5194.725961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5194.734996] LustreError: Skipped 1 previous similar message [ 5195.789349] Lustre: server umount lustre-OST0000 complete [ 5207.016403] LustreError: 157391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.034809] LustreError: 157391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5207.071537] Lustre: server umount lustre-MDT0000 complete [ 5210.894823] LustreError: 158125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156370 with bad export cookie 14887505211755975182 [ 5210.897702] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5210.906632] LustreError: 158125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5221.539984] Lustre: server umount lustre-MDT0001 complete [ 5232.814562] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5236.274961] Key type lgssc unregistered [ 5236.492723] LNet: 163498:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5236.500934] LNetError: 163498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5237.543028] LNet: Removed LNI 192.168.202.136@tcp [ 5238.656193] Key type .llcrypt unregistered [ 5238.658060] Key type ._llcrypt unregistered [ 5263.623228] Key type ._llcrypt registered [ 5263.625220] Key type .llcrypt registered [ 5263.755333] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5264.977340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5265.136665] alg: No test for adler32 (adler32-zlib) [ 5266.189707] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5266.472956] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5268.147447] Key type lgssc registered [ 5269.295196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5278.178564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5278.203062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.762786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5284.443326] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5289.955433] LustreError: 164916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5289.998223] LustreError: 164916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5295.090232] LustreError: 164915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.296832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.917525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5303.409257] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5311.962958] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5318.863917] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5327.760962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5328.058751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.106857] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5344.728473] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5349.631851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5356.006536] Lustre: Failing over lustre-MDT0000 [ 5356.343627] Lustre: server umount lustre-MDT0000 complete [ 5356.946510] LustreError: 164911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.536149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5357.539859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.096656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5360.098496] LustreError: 164915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.123967] Lustre: Skipped 1 previous similar message [ 5360.145478] LustreError: 164915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5365.219759] LustreError: 164911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.263255] LustreError: 164911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5368.840180] LustreError: 164896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.36@tcp arrived at 1788156528 with bad export cookie 8852434555673379076 [ 5375.457597] Lustre: 164228:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156519/real 1788156519] req@00000000748d9e1b x1875017912579072/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788156535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5375.472142] LustreError: 164915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.498635] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.541865] LustreError: 164915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5376.837605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.704682] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7ada2a53804e024d [ 5385.720787] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [ 5386.056213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.129216] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5391.344544] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5391.357649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5392.196670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5395.208862] Lustre: lustre-MDT0000: Denying connection for new client d6c4d81c-2aad-4ae2-89a3-72095e60adc0 (at 192.168.202.36@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5406.692655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5406.721396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5406.740837] Lustre: Skipped 1 previous similar message [ 5411.014705] Lustre: server umount lustre-OST0000 complete [ 5415.183942] LustreError: 168650:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5415.198847] Lustre: 167998:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5415.215922] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5415.235963] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5415.248974] LustreError: 167998:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb2e3f77 x1875017912600192/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5415.266458] LustreError: 167998:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5415.274381] LustreError: 167998:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5415.285934] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5415.679467] Lustre: server umount lustre-MDT0000 complete [ 5416.934374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.947292] Lustre: Skipped 1 previous similar message [ 5416.959400] LustreError: 164912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5416.968679] LustreError: 164912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5419.153629] LustreError: 164895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156578 with bad export cookie 8852434555673379405 [ 5419.159663] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5419.164131] LustreError: 164895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5421.813824] Lustre: server umount lustre-MDT0001 complete [ 5433.535706] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5436.282565] Key type lgssc unregistered [ 5436.654984] LNet: 169403:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5436.670709] LNetError: 169403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5436.696051] LNet: Removed LNI 192.168.202.136@tcp [ 5437.879693] Key type .llcrypt unregistered [ 5437.888625] Key type ._llcrypt unregistered [ 5463.759703] Key type ._llcrypt registered [ 5463.761859] Key type .llcrypt registered [ 5463.858884] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5464.877061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5465.099668] alg: No test for adler32 (adler32-zlib) [ 5466.090674] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5466.424323] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5468.143462] Key type lgssc registered [ 5469.675426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5478.046940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5478.072314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.561453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5484.007960] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5490.153294] LustreError: 170816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.186670] LustreError: 170816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5493.453082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5493.968437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5499.753224] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5509.299969] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5515.651545] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5523.837770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5524.261208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.296101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5531.794805] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5541.114373] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5549.028771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.047580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.055288] Lustre: Skipped 1 previous similar message [ 5554.155089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5554.173605] Lustre: Skipped 1 previous similar message [ 5555.717812] Lustre: server umount lustre-OST0000 complete [ 5564.386902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.421584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5564.438337] Lustre: Skipped 2 previous similar messages [ 5564.471364] Lustre: Skipped 1 previous similar message [ 5568.076531] Lustre: server umount lustre-MDT0000 complete [ 5569.505305] LustreError: 171545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.541909] LustreError: 171545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5574.084195] LustreError: 170798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156733 with bad export cookie 7334129068776891891 [ 5574.098789] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.532104] Lustre: server umount lustre-MDT0001 complete [ 5587.852305] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5591.032440] Key type lgssc unregistered [ 5591.372822] LNet: 174353:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5591.383195] LNetError: 174353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5591.399792] LNet: Removed LNI 192.168.202.136@tcp [ 5592.700861] Key type .llcrypt unregistered [ 5592.702648] Key type ._llcrypt unregistered [ 5631.602376] Key type ._llcrypt registered [ 5631.603860] Key type .llcrypt registered [ 5631.711791] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5632.690487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5632.754205] alg: No test for adler32 (adler32-zlib) [ 5633.925917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5634.176120] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5635.855174] Key type lgssc registered [ 5636.910945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5645.104713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5645.124576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.535890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.913947] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5657.056814] LustreError: 176060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.075617] LustreError: 176060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5658.089225] LustreError: 176059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.959301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5659.185695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5663.898152] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5671.817625] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5678.339065] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5685.532711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5685.958304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.312402] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5693.172825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5698.362504] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5709.793476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5709.799484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5709.815126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5709.830565] Lustre: Skipped 1 previous similar message [ 5709.847149] Lustre: Skipped 1 previous similar message [ 5711.705592] Lustre: server umount lustre-OST0000 complete [ 5720.033579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5720.042466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.049496] Lustre: Skipped 1 previous similar message [ 5722.031591] Lustre: server umount lustre-MDT0000 complete [ 5725.154987] LustreError: 176060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5725.177910] LustreError: 176060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5726.320936] LustreError: 176040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156885 with bad export cookie 14989071227238647864 [ 5726.340879] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5726.341927] LustreError: 176040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5726.899145] Lustre: server umount lustre-MDT0001 complete [ 5739.005375] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5741.430847] Key type lgssc unregistered [ 5741.696934] LNet: 179591:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5741.701993] LNetError: 179591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5741.713863] LNet: Removed LNI 192.168.202.136@tcp [ 5742.431882] Key type .llcrypt unregistered [ 5742.433719] Key type ._llcrypt unregistered [ 5769.237722] Key type ._llcrypt registered [ 5769.240509] Key type .llcrypt registered [ 5769.432474] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5770.998713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5771.178117] alg: No test for adler32 (adler32-zlib) [ 5772.395346] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5772.738319] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5774.544108] Key type lgssc registered [ 5776.118365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.133434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.160545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.693119] Lustre: *** cfs_fail_loc=135, val=0*** [ 5792.825458] LustreError: 181117:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5792.836768] LustreError: 181117:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -2 [ 5792.845787] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5792.845787] [ 5792.853251] LustreError: MGC192.168.202.136@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5792.862735] LustreError: 181078:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5792.872403] LustreError: 181078:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 5792.896691] LustreError: 181078:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 5792.912839] LustreError: 181078:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5793.021685] Lustre: server umount lustre-MDT0000 complete [ 5793.028755] LustreError: 181078:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5801.487598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.790460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5807.118347] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5812.195601] LustreError: 181488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5812.209421] LustreError: 181488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5817.318663] LustreError: 181489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.435065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.096029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5825.523096] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5839.495533] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5846.907387] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.133799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.148208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.884412] Lustre: server umount lustre-MDT0000 complete [ 5857.248977] LustreError: 181485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.265660] LustreError: 181485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5859.550685] LustreError: 181468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157019 with bad export cookie 13591440840938663154 [ 5859.554726] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5859.580223] LustreError: 181468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5859.904376] Lustre: server umount lustre-MDT0001 complete [ 5871.617670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5874.973368] Key type lgssc unregistered [ 5875.346236] LNet: 183813:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5875.361301] LNetError: 183813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5875.384416] LNet: Removed LNI 192.168.202.136@tcp [ 5876.479276] Key type .llcrypt unregistered [ 5876.481307] Key type ._llcrypt unregistered [ 5902.285950] Key type ._llcrypt registered [ 5902.291964] Key type .llcrypt registered [ 5902.421689] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 5902.739477] Key type .llcrypt unregistered [ 5902.742057] Key type ._llcrypt unregistered [ 5921.799520] Key type ._llcrypt registered [ 5921.801822] Key type .llcrypt registered [ 5921.913715] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5923.207201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5923.287819] alg: No test for adler32 (adler32-zlib) [ 5924.447768] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5924.699138] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5926.343681] Key type lgssc registered [ 5927.501854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5935.836304] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5935.866929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5951.455417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5956.575379] LustreError: 185950:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 5983.263251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.083223] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6002.595418] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6013.242846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6014.110994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.236899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6019.957479] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6023.654570] LustreError: 186989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6023.698653] LustreError: 186989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6028.774835] LustreError: 186990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6028.789733] LustreError: 186990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6029.279195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.929439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6035.044123] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6043.409556] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6049.912821] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6052.961877] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6065.121493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6065.136419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.168808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.206823] Lustre: server umount lustre-OST0000 complete [ 6075.882414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.890662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6075.906854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.906865] Lustre: Skipped 1 previous similar message [ 6075.908340] Lustre: Skipped 2 previous similar messages [ 6080.995782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.002053] Lustre: Skipped 1 previous similar message [ 6081.857938] Lustre: server umount lustre-MDT0000 complete [ 6086.114098] LustreError: 186994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.142167] LustreError: 186994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6086.239775] LustreError: 188611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157245 with bad export cookie 6703927735068006946 [ 6086.248150] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6086.261820] LustreError: 188611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6087.270791] Lustre: server umount lustre-MDT0001 complete [ 6101.199242] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6104.164408] Key type lgssc unregistered [ 6104.476734] LNet: 189870:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.494715] LNetError: 189870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.544688] LNet: Removed LNI 192.168.202.136@tcp [ 6105.595680] Key type .llcrypt unregistered [ 6105.597940] Key type ._llcrypt unregistered [ 6136.326711] Key type ._llcrypt registered [ 6136.328846] Key type .llcrypt registered [ 6136.446373] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6138.155810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6138.214780] alg: No test for adler32 (adler32-zlib) [ 6139.434798] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6139.923722] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6141.729323] Key type lgssc registered [ 6142.986690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6151.742334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6151.764363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.143170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6158.110277] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6163.426876] LustreError: 191284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.450406] LustreError: 191284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6167.542396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.883544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6173.264093] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6181.610204] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6189.467320] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.825283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.421267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.486105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6206.443497] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6216.790937] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6222.490483] Lustre: Failing over lustre-MDT0000 [ 6222.829972] Lustre: server umount lustre-MDT0000 complete [ 6223.342256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.349053] LustreError: 191281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.361861] Lustre: Skipped 2 previous similar messages [ 6223.397654] LustreError: 191281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6228.458971] LustreError: 191280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.477527] LustreError: 191280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6233.573057] LustreError: 192012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.596405] LustreError: 192012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6238.696406] Lustre: 190597:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157382/real 1788157382] req@00000000b3550eba x1875018827984896/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788157398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6238.707516] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6238.721644] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6238.784436] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6243.609981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.934137] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.976460] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6249.231673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6249.289422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6250.349423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6253.951087] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6254.583159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6254.601448] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6254.658266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6259.022252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6262.732778] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6278.128218] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6278.137403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.160819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.648892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.667738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.690784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6280.267700] Lustre: server umount lustre-OST0000 complete [ 6285.281246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.284139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.291212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6285.303608] Lustre: Skipped 1 previous similar message [ 6290.401557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.409353] Lustre: Skipped 2 previous similar messages [ 6291.263078] Lustre: server umount lustre-MDT0000 complete [ 6294.506101] LustreError: 191284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.551066] LustreError: 191284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6295.442701] LustreError: 195351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157455 with bad export cookie 5570539667835059656 [ 6295.461635] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6295.472587] LustreError: 195351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6296.172368] Lustre: server umount lustre-MDT0001 complete [ 6311.458914] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6316.308540] Key type lgssc unregistered [ 6316.665804] LNet: 196102:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.682875] LNetError: 196102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6316.715267] LNet: Removed LNI 192.168.202.136@tcp [ 6318.295499] Key type .llcrypt unregistered [ 6318.297447] Key type ._llcrypt unregistered [ 6350.333464] Key type ._llcrypt registered [ 6350.336805] Key type .llcrypt registered [ 6350.462734] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6351.676469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6351.692233] alg: No test for adler32 (adler32-zlib) [ 6352.751443] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6353.072768] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6354.903318] Key type lgssc registered [ 6356.530900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6366.176671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6366.217324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6367.899229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.262847] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6377.456896] LustreError: 197533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.505632] LustreError: 197533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6382.560330] LustreError: 197532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.369484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.671438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6389.052582] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6398.159769] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6405.115527] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6412.514815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.855043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6413.902651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6419.398416] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6428.325369] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.582164] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6439.670392] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6457.627925] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6470.113053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6470.116230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.126838] Lustre: Skipped 1 previous similar message [ 6470.128421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6470.150473] Lustre: Skipped 1 previous similar message [ 6474.203545] Lustre: server umount lustre-OST0000 complete [ 6480.356352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.380384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6480.385450] Lustre: Skipped 1 previous similar message [ 6485.474983] LustreError: 197533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.488491] LustreError: 197533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6485.514919] Lustre: server umount lustre-MDT0000 complete [ 6489.526280] LustreError: 197514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157649 with bad export cookie 4245674426496924442 [ 6489.530839] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.537298] LustreError: 197514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6490.110942] Lustre: server umount lustre-MDT0001 complete [ 6505.560857] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6508.922914] Key type lgssc unregistered [ 6509.244703] LNet: 201403:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6509.252961] LNetError: 201403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6510.310677] LNet: Removed LNI 192.168.202.136@tcp [ 6511.050983] Key type .llcrypt unregistered [ 6511.052937] Key type ._llcrypt unregistered [ 6533.558476] Key type ._llcrypt registered [ 6533.560876] Key type .llcrypt registered [ 6533.671428] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6534.789303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6534.820436] alg: No test for adler32 (adler32-zlib) [ 6535.970583] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6536.204844] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6537.927251] Key type lgssc registered [ 6538.853806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6547.886832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6547.944397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6549.755744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.620401] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6560.233434] LustreError: 202672:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.260641] LustreError: 202672:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6563.964629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.563311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6570.481182] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6578.358206] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6584.354740] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6592.650367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6593.192246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6595.266065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6599.108637] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6607.345818] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.847818] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6624.034716] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6632.096521] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6635.964653] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6651.887677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6651.889991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.909501] LustreError: Skipped 1 previous similar message [ 6651.931516] Lustre: Skipped 1 previous similar message [ 6651.941740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6651.952914] Lustre: Skipped 1 previous similar message [ 6654.727175] Lustre: server umount lustre-OST0000 complete [ 6661.601914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6661.619587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.646642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.181734] Lustre: server umount lustre-MDT0000 complete [ 6667.233418] LustreError: 202667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.249970] LustreError: 202667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6669.299668] LustreError: 202653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157828 with bad export cookie 10874142954761153569 [ 6669.302304] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6669.309231] LustreError: 202653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6669.603992] Lustre: server umount lustre-MDT0001 complete [ 6682.771287] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6685.186331] Key type lgssc unregistered [ 6685.438193] LNet: 207456:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6685.448377] LNetError: 207456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6685.465592] LNet: Removed LNI 192.168.202.136@tcp [ 6686.159400] Key type .llcrypt unregistered [ 6686.161614] Key type ._llcrypt unregistered [ 6715.687583] Key type ._llcrypt registered [ 6715.689929] Key type .llcrypt registered [ 6715.807632] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6716.675898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6716.698245] alg: No test for adler32 (adler32-zlib) [ 6717.813962] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6718.084776] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6719.759748] Key type lgssc registered [ 6721.421274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6730.989698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6731.013272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.594755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6738.378615] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6743.010587] LustreError: 208870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.046705] LustreError: 208870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6748.129126] LustreError: 208869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.200744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.773329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6753.687626] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6762.886338] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6769.925798] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.986230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6778.274914] LustreError: 210691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.474811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.485615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6785.240813] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6793.823837] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6814.194746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.196870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6814.205392] Lustre: Skipped 1 previous similar message [ 6814.225218] Lustre: Skipped 1 previous similar message [ 6816.187467] Lustre: server umount lustre-OST0000 complete [ 6820.321604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6820.326901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.337113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.417396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.417623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6826.512394] Lustre: server umount lustre-MDT0000 complete [ 6829.543210] LustreError: 208866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6829.569251] LustreError: 208866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6830.627943] LustreError: 209604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157990 with bad export cookie 18148973066878450336 [ 6830.641826] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6830.652843] LustreError: 209604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6831.011891] Lustre: server umount lustre-MDT0001 complete [ 6844.760215] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6848.735534] Key type lgssc unregistered [ 6849.047949] LNet: 212554:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6849.054605] LNetError: 212554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6849.075550] LNet: Removed LNI 192.168.202.136@tcp [ 6850.432912] Key type .llcrypt unregistered [ 6850.437898] Key type ._llcrypt unregistered [ 6881.303575] Key type ._llcrypt registered [ 6881.306401] Key type .llcrypt registered [ 6881.459292] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6882.819721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6882.995894] alg: No test for adler32 (adler32-zlib) [ 6884.111716] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6884.427627] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6886.127203] Key type lgssc registered [ 6887.713069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.965717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6896.999647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.514897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6903.131511] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6908.900296] LustreError: 213838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6908.926309] LustreError: 213838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6913.064144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6913.710514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6919.243364] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6927.849961] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6934.396684] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6942.056078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6942.534577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.609853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6948.915120] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6957.646191] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6974.437897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.440137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6974.446367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6974.453950] Lustre: Skipped 1 previous similar message [ 6974.485621] LustreError: Skipped 1 previous similar message [ 6978.270455] Lustre: server umount lustre-OST0000 complete [ 6984.684215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.694532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.697947] Lustre: Skipped 1 previous similar message [ 6987.695622] Lustre: server umount lustre-MDT0000 complete [ 6989.792677] LustreError: 213838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.921756] LustreError: 213818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158151 with bad export cookie 15581799715114604460 [ 6991.941082] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.950530] LustreError: 213818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6992.318857] Lustre: server umount lustre-MDT0001 complete [ 7005.001298] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7007.573398] Key type lgssc unregistered [ 7007.889996] LNet: 217520:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7007.895510] LNetError: 217520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7008.934912] LNet: Removed LNI 192.168.202.136@tcp [ 7010.186479] Key type .llcrypt unregistered [ 7010.188323] Key type ._llcrypt unregistered [ 7031.855797] Key type ._llcrypt registered [ 7031.859899] Key type .llcrypt registered [ 7031.952993] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7033.172648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7033.277650] alg: No test for adler32 (adler32-zlib) [ 7034.366744] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7034.553119] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7036.184411] Key type lgssc registered [ 7037.581575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7045.644316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7045.657557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.029711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7051.032177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7057.378368] LustreError: 218791:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.395148] LustreError: 218791:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7059.852120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7060.310785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7065.288594] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7073.069308] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7078.441308] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7084.972135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.376399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7091.525383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7093.733500] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7102.984297] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.310297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.421549] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7113.686504] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7120.118815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7128.810753] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7130.646764] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7142.726982] Lustre: Permanently deactivating lustre-OST0001 [ 7142.743359] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7151.754450] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7158.405921] Lustre: Permanently reactivating lustre-OST0001 [ 7158.428716] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7158.436759] Lustre: Skipped 2 previous similar messages [ 7164.512362] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7164.522957] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7164.530525] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7164.540165] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7164.564399] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7169.002030] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.021337] Lustre: Skipped 1 previous similar message [ 7169.030176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7169.409777] Lustre: server umount lustre-OST0001 complete [ 7173.600622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7173.619171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7173.640149] Lustre: Skipped 1 previous similar message [ 7173.659750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7173.671379] Lustre: Skipped 1 previous similar message [ 7174.116541] LustreError: 220614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7174.158204] LustreError: 220614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7179.236824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7179.239661] LustreError: 220614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.251264] Lustre: Skipped 3 previous similar messages [ 7179.272815] LustreError: 220614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7179.668604] Lustre: server umount lustre-OST0000 complete [ 7183.329278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7183.335432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7183.353589] Lustre: Skipped 1 previous similar message [ 7183.358596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.472235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.483015] Lustre: Skipped 3 previous similar messages [ 7189.594741] Lustre: server umount lustre-MDT0000 complete [ 7192.714343] LustreError: 218771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158352 with bad export cookie 17259268234175861346 [ 7192.716244] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7192.726311] LustreError: 218771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7193.173965] Lustre: server umount lustre-MDT0001 complete [ 7203.771670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7206.706368] Key type lgssc unregistered [ 7207.009834] LNet: 223834:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7207.014990] LNetError: 223834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7207.028823] LNet: Removed LNI 192.168.202.136@tcp [ 7207.620509] Key type .llcrypt unregistered [ 7207.624583] Key type ._llcrypt unregistered [ 7242.118732] Key type ._llcrypt registered [ 7242.120907] Key type .llcrypt registered [ 7242.209300] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7243.526882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7243.544660] alg: No test for adler32 (adler32-zlib) [ 7244.639384] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7244.907923] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7246.567286] Key type lgssc registered [ 7247.666796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7257.420097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7257.481573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.973046] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7259.019455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7259.301325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.226523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7274.328308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.477375] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7274.533564] Lustre: 226031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7274.817948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7279.967222] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7287.727985] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7294.381946] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7299.990559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.114669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7300.291353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.542759] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7307.929805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7312.497106] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.476845] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7323.458331] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7330.494917] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7338.308307] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7347.344143] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7365.034644] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7365.044857] Lustre: Skipped 1 previous similar message [ 7397.880686] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7397.887816] Lustre: Skipped 3 previous similar messages [ 7464.674388] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7464.683491] Lustre: Skipped 7 previous similar messages [ 7502.304495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7502.311761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7502.334072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7502.819699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7502.828936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7502.849228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7506.849855] Lustre: server umount lustre-OST0000 complete [ 7513.057891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.063883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7513.101014] Lustre: Skipped 1 previous similar message [ 7513.120225] Lustre: Skipped 1 previous similar message [ 7516.871781] Lustre: server umount lustre-MDT0000 complete [ 7520.221386] LustreError: 226025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158679 with bad export cookie 16466622564973389576 [ 7520.223564] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.228628] LustreError: 226025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7520.520175] Lustre: server umount lustre-MDT0001 complete [ 7533.838387] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7537.006614] Key type lgssc unregistered [ 7537.392809] LNet: 230837:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7537.397831] LNetError: 230837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7537.412505] LNet: Removed LNI 192.168.202.136@tcp [ 7538.080740] Key type .llcrypt unregistered [ 7538.082440] Key type ._llcrypt unregistered [ 7565.274855] Key type ._llcrypt registered [ 7565.276527] Key type .llcrypt registered [ 7565.401550] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7566.524112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7566.629989] alg: No test for adler32 (adler32-zlib) [ 7567.639217] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7567.859287] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7569.551193] Key type lgssc registered [ 7570.565306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7578.034861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7578.092805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7579.606584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.100681] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7589.858225] LustreError: 232250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7589.887240] LustreError: 232250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7593.777225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7594.295042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7599.315286] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7607.508380] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7613.714924] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7620.928264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7621.563494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7624.648887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7628.530261] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7637.759298] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7654.368084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7654.375451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7654.380935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7654.406678] Lustre: Skipped 1 previous similar message [ 7659.492440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7659.507470] Lustre: Skipped 2 previous similar messages [ 7664.614133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7664.626540] Lustre: Skipped 1 previous similar message [ 7666.143197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7666.405947] Lustre: server umount lustre-OST0000 complete [ 7671.265734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7671.280705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.298809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.854451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.695470] Lustre: server umount lustre-MDT0000 complete [ 7679.970895] LustreError: 232250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.996250] LustreError: 232250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7680.442902] LustreError: 232232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158840 with bad export cookie 9383860111192100594 [ 7680.448409] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7680.456910] LustreError: 232232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7680.678189] Lustre: server umount lustre-MDT0001 complete [ 7692.342662] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7695.116923] Key type lgssc unregistered [ 7695.428682] LNet: 235943:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7695.436662] LNetError: 235943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7695.451229] LNet: Removed LNI 192.168.202.136@tcp [ 7696.203397] Key type .llcrypt unregistered [ 7696.205636] Key type ._llcrypt unregistered [ 7720.931502] Key type ._llcrypt registered [ 7720.935331] Key type .llcrypt registered [ 7721.093712] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7722.679616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7722.752913] alg: No test for adler32 (adler32-zlib) [ 7723.798658] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7723.990979] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7725.663202] Key type lgssc registered [ 7727.359540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7996.575805] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7999.206530] Key type lgssc unregistered [ 7999.507253] LNet: 237966:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7999.516586] LNetError: 237966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7999.550199] LNet: Removed LNI 192.168.202.136@tcp [ 8000.363809] Key type .llcrypt unregistered [ 8000.366688] Key type ._llcrypt unregistered [ 8028.920773] Key type ._llcrypt registered [ 8028.923192] Key type .llcrypt registered [ 8029.014724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8046.870308] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8047.915280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8047.926265] alg: No test for adler32 (adler32-zlib) [ 8049.123751] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8049.312974] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8050.943372] Key type lgssc registered [ 8052.102498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8084.090855] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7817 sec ========== 02:54:01 (1788159241) [ 8086.135773] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:54:03 (1788159243) === [ 8090.673141] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:54:07 (1788159247) === [ 8135.228814] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8137.650461] Key type lgssc unregistered [ 8137.876816] LNet: 243203:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8137.880768] LNetError: 243203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8137.891919] LNet: Removed LNI 192.168.202.136@tcp [ 8138.583832] Key type .llcrypt unregistered [ 8138.588395] Key type ._llcrypt unregistered