[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431469192 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002240] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011080] LSM: Security Framework initializing [ 0.012044] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014090] *** VALIDATE selinux *** [ 0.022502] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027580] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029012] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031084] *** VALIDATE tmpfs *** [ 0.032494] *** VALIDATE proc *** [ 0.034248] *** VALIDATE cgroup *** [ 0.035005] *** VALIDATE cgroup2 *** [ 0.036254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039021] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.043003] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.045175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046424] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047009] ... version: 2 [ 0.047746] ... bit width: 48 [ 0.048005] ... generic registers: 4 [ 0.048866] ... value mask: 0000ffffffffffff [ 0.049005] ... max period: 00007fffffffffff [ 0.050005] ... fixed-purpose events: 3 [ 0.050855] ... event mask: 000000070000000f [ 0.052128] rcu: Hierarchical SRCU implementation. [ 0.054086] smp: Bringing up secondary CPUs ... [ 0.055382] x86: Booting SMP configuration: [ 0.056014] .... node #0, CPUs: #1 #2 #3 [ 0.058730] smp: Brought up 1 node, 4 CPUs [ 0.059914] smpboot: Max logical packages: 1 [ 0.060006] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.240848] node 0 deferred pages initialised in 179ms [ 0.243232] devtmpfs: initialized [ 0.244216] x86/mm: Memory block size: 128MB [ 0.246953] gcov: version magic: 0x41383552 [ 0.248128] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.250217] pinctrl core: initialized pinctrl subsystem [ 0.251152] [ 0.251747] ************************************************************* [ 0.252009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253007] ** ** [ 0.254007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255005] ** ** [ 0.256005] ** This means that this kernel is built to expose internal ** [ 0.257005] ** IOMMU data structures, which may compromise security on ** [ 0.258005] ** your system. ** [ 0.259005] ** ** [ 0.260005] ** If you see this message and you are not debugging the ** [ 0.261005] ** kernel, report this immediately to your vendor! ** [ 0.262006] ** ** [ 0.263005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264005] ************************************************************* [ 0.265484] NET: Registered protocol family 16 [ 0.266362] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.267025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.269303] cpuidle: using governor menu [ 0.271513] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273496] PCI: Using configuration type 1 for base access [ 0.276114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283008] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.284094] cryptd: max_cpu_qlen set to 1000 [ 0.285167] ACPI: Added _OSI(Module Device) [ 0.286016] ACPI: Added _OSI(Processor Device) [ 0.287000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.291259] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.296240] ACPI: Interpreter enabled [ 0.297038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.297910] ACPI: Using IOAPIC for interrupt routing [ 0.299048] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301246] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.308607] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.310017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.314036] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.319079] acpiphp: Slot [2] registered [ 0.320099] acpiphp: Slot [5] registered [ 0.322108] acpiphp: Slot [6] registered [ 0.323070] acpiphp: Slot [7] registered [ 0.324073] acpiphp: Slot [8] registered [ 0.326083] acpiphp: Slot [9] registered [ 0.327067] acpiphp: Slot [10] registered [ 0.329068] acpiphp: Slot [3] registered [ 0.330079] acpiphp: Slot [4] registered [ 0.332067] acpiphp: Slot [11] registered [ 0.334106] acpiphp: Slot [12] registered [ 0.335066] acpiphp: Slot [13] registered [ 0.337070] acpiphp: Slot [14] registered [ 0.338088] acpiphp: Slot [15] registered [ 0.340069] acpiphp: Slot [16] registered [ 0.341085] acpiphp: Slot [17] registered [ 0.343088] acpiphp: Slot [18] registered [ 0.345068] acpiphp: Slot [19] registered [ 0.347068] acpiphp: Slot [20] registered [ 0.348066] acpiphp: Slot [21] registered [ 0.350097] acpiphp: Slot [22] registered [ 0.351087] acpiphp: Slot [23] registered [ 0.353067] acpiphp: Slot [24] registered [ 0.355069] acpiphp: Slot [25] registered [ 0.356067] acpiphp: Slot [26] registered [ 0.358082] acpiphp: Slot [27] registered [ 0.359072] acpiphp: Slot [28] registered [ 0.361067] acpiphp: Slot [29] registered [ 0.363106] acpiphp: Slot [30] registered [ 0.364089] acpiphp: Slot [31] registered [ 0.365053] PCI host bridge to bus 0000:00 [ 0.365847] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.367009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.369008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.370007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.372008] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.374010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.375139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376817] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379962] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.387541] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.393080] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.396010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.398008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.403609] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406885] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.410029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.413840] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.417946] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.434011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440377] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.445009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472291] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.480008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.491011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498751] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.508013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.521016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529445] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.544013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.560016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.568815] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.573899] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.578010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.591013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.598901] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.603012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.607009] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.619012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.629000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.631328] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.633301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635204] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.636130] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.641029] iommu: Default domain type: Passthrough [ 0.642504] SCSI subsystem initialized [ 0.644108] ACPI: bus type USB registered [ 0.646095] usbcore: registered new interface driver usbfs [ 0.648056] usbcore: registered new interface driver hub [ 0.650061] usbcore: registered new device driver usb [ 0.652164] pps_core: LinuxPPS API ver. 1 registered [ 0.654012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.657052] PTP clock support registered [ 0.659132] EDAC MC: Ver: 3.0.0 [ 0.661106] PCI: Using ACPI for IRQ routing [ 0.662627] NetLabel: Initializing [ 0.663007] NetLabel: domain hash size = 128 [ 0.664008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.666060] NetLabel: unlabeled traffic allowed by default [ 0.668186] vgaarb: loaded [ 0.669235] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.671011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676375] clocksource: Switched to clocksource kvm-clock [ 0.763826] VFS: Disk quotas dquot_6.6.0 [ 0.765303] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.767975] *** VALIDATE ramfs *** [ 0.769278] *** VALIDATE hugetlbfs *** [ 0.770780] pnp: PnP ACPI init [ 0.773191] pnp: PnP ACPI: found 6 devices [ 0.788355] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.791652] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.793751] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.795974] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.798320] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.800613] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.803283] NET: Registered protocol family 2 [ 0.805693] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.810601] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.814067] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820085] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.822621] TCP: Hash tables configured (established 65536 bind 65536) [ 0.824896] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.826879] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.828694] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.830597] NET: Registered protocol family 1 [ 0.832272] RPC: Registered named UNIX socket transport module. [ 0.834554] RPC: Registered udp transport module. [ 0.836187] RPC: Registered tcp transport module. [ 0.837970] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.840412] NET: Registered protocol family 44 [ 0.841570] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.843032] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.844650] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.847107] PCI: CLS 0 bytes, default 64 [ 0.848334] Unpacking initramfs... [ 2.210653] debug: unmapping init [mem 0xffff9e4ffcc54000-0xffff9e4ffffbffff] [ 2.214333] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.216353] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.220368] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.643517] Initialise system trusted keyrings [ 2.644451] Key type blacklist registered [ 2.645602] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.655046] zbud: loaded [ 2.657529] *** VALIDATE nfs *** [ 2.658534] *** VALIDATE nfs4 *** [ 2.659864] pstore: using deflate compression [ 2.662146] Platform Keyring initialized [ 2.731656] NET: Registered protocol family 38 [ 2.733349] Key type asymmetric registered [ 2.734680] Asymmetric key parser 'x509' registered [ 2.736234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.738587] io scheduler mq-deadline registered [ 2.739821] io scheduler kyber registered [ 2.741129] io scheduler bfq registered [ 2.743044] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.745799] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.747673] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.750536] ACPI: Power Button [PWRF] [ 2.755238] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.760255] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.769103] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.773835] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.784602] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.811868] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.839256] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.843891] Non-volatile memory driver v1.3 [ 2.844989] Linux agpgart interface v0.103 [ 2.869413] virtio_blk virtio1: [vda] 134816 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.871811] vda: detected capacity change from 0 to 69025792 [ 2.883681] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.886572] vdb: detected capacity change from 0 to 1073741824 [ 2.897781] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.899591] vdc: detected capacity change from 0 to 2621440000 [ 2.912023] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.914549] vdd: detected capacity change from 0 to 2621440000 [ 2.930270] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.932394] vde: detected capacity change from 0 to 4294967296 [ 2.943123] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.945725] vdf: detected capacity change from 0 to 4294967296 [ 2.951781] libphy: Fixed MDIO Bus: probed [ 2.956190] usbcore: registered new interface driver usbserial_generic [ 2.958524] usbserial: USB Serial support registered for generic [ 2.960751] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.964806] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.966715] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.968955] mousedev: PS/2 mouse device common for all mice [ 2.971606] rtc_cmos 00:05: RTC can wake from S4 [ 2.973800] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.977336] rtc_cmos 00:05: registered as rtc0 [ 2.978179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.979340] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.983483] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.984587] intel_pstate: CPU model not supported [ 2.990893] hid: raw HID events driver (C) Jiri Kosina [ 2.992900] usbcore: registered new interface driver usbhid [ 2.994647] usbhid: USB HID core driver [ 2.995853] drop_monitor: Initializing network drop monitor service [ 2.997859] Initializing XFRM netlink socket [ 2.999615] NET: Registered protocol family 10 [ 3.002200] Segment Routing with IPv6 [ 3.003251] NET: Registered protocol family 17 [ 3.004506] mpls_gso: MPLS GSO support [ 3.009939] RAS: Correctable Errors collector initialized. [ 3.011794] AVX version of gcm_enc/dec engaged. [ 3.012887] AES CTR mode by8 optimization enabled [ 3.076179] sched_clock: Marking stable (3076016512, 0)->(3770077232, -694060720) [ 3.079406] registered taskstats version 1 [ 3.080778] Loading compiled-in X.509 certificates [ 3.082584] zswap: loaded using pool lzo/zbud [ 3.102404] Key type big_key registered [ 3.113799] Key type encrypted registered [ 3.115097] ima: No TPM chip found, activating TPM-bypass! [ 3.116717] ima: Allocated hash algorithm: sha1 [ 3.118371] ima: No architecture policies found [ 3.119934] evm: Initialising EVM extended attributes: [ 3.121706] evm: security.selinux [ 3.122685] evm: security.ima [ 3.123430] evm: security.capability [ 3.124869] evm: HMAC attrs: 0x1 [ 3.128337] rtc_cmos 00:05: setting system clock to 2026-04-30 23:49:08 UTC (1777592948) [ 3.134726] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 3.137599] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 3.142061] Write protecting the kernel read-only data: 28672k [ 3.145167] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 3.147948] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 3.181420] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.189552] systemd[1]: Detected virtualization kvm. [ 3.191392] systemd[1]: Detected architecture x86-64. [ 3.192855] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.220238] systemd[1]: No hostname configured. [ 3.221785] systemd[1]: Set hostname to . [ 3.224974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.227155] systemd[1]: Initializing machine ID from random generator. [ 3.261861] random: ln: uninitialized urandom read (6 bytes read) [ 3.352658] random: systemd: uninitialized urandom read (16 bytes read) [ 3.354550] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.361459] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.365801] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.924472] device-mapper: uevent: version 1.0.3 [ 3.927237] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.737691] virtio_net virtio0 ens2: renamed from eth0 [ 4.740827] scsi host0: ata_piix [ 4.774400] random: fast init done [ 4.840104] scsi host1: ata_piix [ 4.841956] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.847642] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.633223] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.658642] random: crng init done [ 9.662733] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.405529] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.917726] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.333664] SELinux: Disabled at runtime. [ 12.405619] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.414903] systemd[1]: Detected virtualization kvm. [ 12.416595] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.101042] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.104657] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.110632] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.114871] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.118689] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.129544] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.134976] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK [[ 13.291051] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.831380] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.302955] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.408341] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.556519] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.575795] EDAC sbridge: Ver: 1.1.2 [ 16.536735] Key type dns_resolver registered [ 16.928501] NFS: Registering the id_resolver key type [ 16.930439] Key type id_resolver registered [ 16.932433] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 39.729539] libcfs: loading out-of-tree module taints kernel. [ 39.745696] Key type ._llcrypt registered [ 39.747505] Key type .llcrypt registered [ 39.783372] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 46.309107] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 46.773175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 46.779589] alg: No test for adler32 (adler32-zlib) [ 47.722852] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 47.958169] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 49.559168] Key type lgssc registered [ 50.057448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 55.619490] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.522280] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 75.963697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.984410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 77.133815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 77.158193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 77.228688] Lustre: lustre-MDT0000: new disk, initializing [ 77.277370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 77.288317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 79.760703] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.311834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 88.370508] Lustre: 6478:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 88.388984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 88.392303] Lustre: Skipped 1 previous similar message [ 88.452715] Lustre: lustre-MDT0001: new disk, initializing [ 88.506019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 88.533474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 88.542217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 91.313086] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.609533] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 100.059471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 100.208608] Lustre: lustre-OST0000: new disk, initializing [ 100.214816] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 100.255276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.691687] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.931969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.935473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.970778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 112.075421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 112.160620] Lustre: lustre-OST0001: new disk, initializing [ 112.162804] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 112.198493] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.404407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 113.413133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 113.460090] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.920268] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 123.594831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.794029] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 131.230167] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 132.998917] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 134.842140] Lustre: DEBUG MARKER: Client: 2.17.52.126 [ 135.929188] Lustre: DEBUG MARKER: MDS: 2.17.52.126 [ 136.935217] Lustre: DEBUG MARKER: OSS: 2.17.52.126 [ 137.617879] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Apr 30 19:51:21 EDT 2026 [ 144.709778] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 145.317269] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 159.711973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 159.712524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.719082] Lustre: Skipped 2 previous similar messages [ 159.725604] Lustre: Skipped 1 previous similar message [ 163.447950] Lustre: server umount lustre-MDT0000 complete [ 165.031639] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593110 with bad export cookie 14896812848141529360 [ 165.033767] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 165.037615] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 169.951952] LustreError: 6483:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 169.952177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.953379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 169.953384] Lustre: Skipped 2 previous similar messages [ 169.960216] LustreError: 6483:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 169.977282] Lustre: Skipped 2 previous similar messages [ 170.976524] LustreError: 6483:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 170.976938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.990890] LustreError: 6483:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 170.995370] Lustre: Skipped 2 previous similar messages [ 171.234738] Lustre: server umount lustre-MDT0001 complete [ 187.360154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 187.400818] Lustre: server umount lustre-OST0000 complete [ 195.169358] Lustre: server umount lustre-OST0001 complete [ 197.600230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 201.494349] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 217.096501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.241598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.268244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 217.319623] Lustre: lustre-MDT0000: new disk, initializing [ 217.365825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 217.380364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.169292] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 224.796059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.853873] Lustre: 14941:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.987499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.716614] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 230.282267] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 231.704014] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 234.611203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.744623] Lustre: lustre-OST0000: new disk, initializing [ 234.746564] Lustre: Skipped 1 previous similar message [ 234.749130] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.751292] Lustre: Skipped 2 previous similar messages [ 234.793094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.797400] Lustre: Skipped 1 previous similar message [ 236.430730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.434656] Lustre: Skipped 1 previous similar message [ 236.439963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.492042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.243400] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 240.883683] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 242.620623] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 242.713183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 244.376568] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 244.457074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 246.753562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 246.753631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.756168] LustreError: Skipped 1 previous similar message [ 246.756851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 246.760215] Lustre: Skipped 1 previous similar message [ 246.767875] Lustre: Skipped 1 previous similar message [ 251.357536] Lustre: server umount lustre-OST0000 complete [ 255.968295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 255.971040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.976951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.231115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 267.232315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.237804] Lustre: Skipped 5 previous similar messages [ 267.302855] Lustre: server umount lustre-MDT0000 complete [ 268.625646] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593213 with bad export cookie 14896812848141530893 [ 268.627649] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.630948] LustreError: 14935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 268.766291] Lustre: server umount lustre-MDT0001 complete [ 269.977429] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:53:34 (1777593214) [ 272.259070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 272.409215] LustreError: 18800:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.434179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.749513] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 277.099738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.904870] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 281.252167] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 282.969804] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 285.807938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.623986] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 291.312221] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 307.683055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 307.683113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.687808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 307.693928] Lustre: Skipped 2 previous similar messages [ 307.698400] Lustre: Skipped 1 previous similar message [ 311.331185] Lustre: server umount lustre-OST0000 complete [ 313.312091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.315913] LustreError: Skipped 1 previous similar message [ 317.923233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 317.929935] Lustre: Skipped 1 previous similar message [ 319.146645] Lustre: server umount lustre-MDT0000 complete [ 320.701613] LustreError: 18827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593266 with bad export cookie 14896812848141531775 [ 320.704502] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.707618] LustreError: 18827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.843452] Lustre: server umount lustre-MDT0001 complete [ 325.308469] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 326.577776] Key type lgssc unregistered [ 326.722562] LNet: 21991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.727274] LNetError: 21991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.740840] LNet: Removed LNI 192.168.204.131@tcp [ 327.143363] Key type .llcrypt unregistered [ 327.145198] Key type ._llcrypt unregistered [ 337.475141] Key type ._llcrypt registered [ 337.476945] Key type .llcrypt registered [ 337.523376] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 337.921950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.986376] alg: No test for adler32 (adler32-zlib) [ 338.863455] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 338.966399] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 340.559205] Key type lgssc registered [ 341.050825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 344.265448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 344.277100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.512645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.094587] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 350.760049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.921340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 352.457595] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 354.730725] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 356.330232] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 359.158141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 359.337215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 361.729915] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 364.171594] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 370.599160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 375.780342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 375.780597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 375.789649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 375.795232] Lustre: Skipped 1 previous similar message [ 377.050650] Lustre: server umount lustre-OST0000 complete [ 380.702974] Lustre: server umount lustre-MDT0000 complete [ 380.898146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.899988] LustreError: 23365:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.905031] Lustre: Skipped 2 previous similar messages [ 380.912891] LustreError: 23365:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 382.202277] LustreError: 24983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593327 with bad export cookie 12427335371243254911 [ 382.205086] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 382.208822] LustreError: 24983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 382.335731] Lustre: server umount lustre-MDT0001 complete [ 386.651588] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 387.889123] Key type lgssc unregistered [ 388.032617] LNet: 26980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.036301] LNetError: 26980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.045629] LNet: Removed LNI 192.168.204.131@tcp [ 388.418377] Key type .llcrypt unregistered [ 388.420219] Key type ._llcrypt unregistered [ 399.130303] Key type ._llcrypt registered [ 399.131976] Key type .llcrypt registered [ 399.179814] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 399.565350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 399.574105] alg: No test for adler32 (adler32-zlib) [ 400.448690] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 400.545380] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 402.135163] Key type lgssc registered [ 402.584647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 405.680405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 405.688751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.887542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.416361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 411.936689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.108599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.694220] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 415.926426] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 417.562182] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 423.000984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.218481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.892410] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 428.430144] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 432.424804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 446.945668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 446.949045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.954445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.970236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.970436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.971787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 447.982611] Lustre: Skipped 1 previous similar message [ 453.088355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 453.091579] Lustre: Skipped 1 previous similar message [ 457.183131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 457.241385] Lustre: server umount lustre-OST0000 complete [ 463.328538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.328722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 463.329350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 463.332477] Lustre: Skipped 1 previous similar message [ 464.795307] Lustre: server umount lustre-MDT0000 complete [ 466.091992] LustreError: 28347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593411 with bad export cookie 11553515597774324608 [ 466.095284] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 466.098072] LustreError: 28347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 466.217152] Lustre: server umount lustre-MDT0001 complete [ 470.316708] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 471.466830] Key type lgssc unregistered [ 471.601543] LNet: 31907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.604639] LNetError: 31907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 471.614514] LNet: Removed LNI 192.168.204.131@tcp [ 471.950833] Key type .llcrypt unregistered [ 471.952534] Key type ._llcrypt unregistered [ 482.155309] Key type ._llcrypt registered [ 482.156627] Key type .llcrypt registered [ 482.196676] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 482.593957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 482.603906] alg: No test for adler32 (adler32-zlib) [ 483.468302] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 483.559857] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 485.159170] Key type lgssc registered [ 485.587891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.504850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.512277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.688671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.223115] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 494.642289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.789781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.262759] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 498.256972] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 499.610240] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 502.146236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 502.292647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.458062] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 506.480601] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.620926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 520.676043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 520.676177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.676928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.678943] LustreError: Skipped 1 previous similar message [ 520.688910] Lustre: Skipped 1 previous similar message [ 522.022548] Lustre: server umount lustre-OST0000 complete [ 525.658769] Lustre: server umount lustre-MDT0000 complete [ 525.793569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.795148] LustreError: 33275:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.795170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.800051] Lustre: Skipped 1 previous similar message [ 525.807674] LustreError: 33275:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 527.138329] LustreError: 33262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593472 with bad export cookie 14010727437613817501 [ 527.141523] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.144309] LustreError: 33262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 527.269518] Lustre: server umount lustre-MDT0001 complete [ 531.312559] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 532.444171] Key type lgssc unregistered [ 532.581401] LNet: 36471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.585711] LNetError: 36471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.594304] LNet: Removed LNI 192.168.204.131@tcp [ 532.921049] Key type .llcrypt unregistered [ 532.922500] Key type ._llcrypt unregistered [ 542.636130] Key type ._llcrypt registered [ 542.637595] Key type .llcrypt registered [ 542.686832] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 543.038443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 543.049695] alg: No test for adler32 (adler32-zlib) [ 543.920618] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 544.016713] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 545.607662] Key type lgssc registered [ 546.046979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 549.044930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 549.052886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.252398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 551.692932] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 554.937961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.084073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 556.551650] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 558.644514] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 560.112516] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 562.646161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 562.775516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.880939] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 566.900070] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 576.102418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 580.576741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 580.581814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.589336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.601356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 581.601571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 581.603620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 582.634550] Lustre: lustre-OST0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 582.637285] Lustre: Skipped 1 previous similar message [ 586.721321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.723650] Lustre: Skipped 1 previous similar message [ 591.849173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.857502] Lustre: Skipped 2 previous similar messages [ 594.399155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 594.473958] Lustre: server umount lustre-OST0000 complete [ 611.296030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 611.299708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.305570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.307618] Lustre: Skipped 1 previous similar message [ 613.344943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.173793] Lustre: server umount lustre-MDT0000 complete [ 615.642299] LustreError: 38561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593561 with bad export cookie 17536558458463054251 [ 615.647789] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.648023] LustreError: 38561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 621.811343] Lustre: server umount lustre-MDT0001 complete [ 625.949412] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 627.025383] Key type lgssc unregistered [ 627.162452] LNet: 41213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.166653] LNetError: 41213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.176469] LNet: Removed LNI 192.168.204.131@tcp [ 627.501619] Key type .llcrypt unregistered [ 627.503272] Key type ._llcrypt unregistered [ 637.812313] Key type ._llcrypt registered [ 637.813700] Key type .llcrypt registered [ 637.856695] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 638.211807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.255075] alg: No test for adler32 (adler32-zlib) [ 639.141212] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 639.242406] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 640.831179] Key type lgssc registered [ 641.287436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.592840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 644.603986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.821823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.373872] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 650.665848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.816851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 652.410282] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 654.594827] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 656.118776] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 658.914556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.093732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 661.302796] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 663.343915] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.423130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 676.319903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.324055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.330722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.857733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 677.858672] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.863070] Lustre: Skipped 1 previous similar message [ 677.868935] Lustre: Skipped 1 previous similar message [ 678.885931] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 682.327880] Lustre: server umount lustre-MDT0000 complete [ 682.977289] LustreError: 42606:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.983653] LustreError: 42606:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 683.732226] LustreError: 44205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593629 with bad export cookie 10190696772094658098 [ 683.734957] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.737699] LustreError: 44205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 683.997047] LustreError: 42582:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.006763] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 688.096940] LustreError: 42583:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 688.097323] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.112652] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 693.215838] LustreError: 42584:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 693.222876] LustreError: 42584:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 698.335163] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 698.432386] Lustre: server umount lustre-MDT0001 complete [ 716.758680] Lustre: server umount lustre-OST0000 complete [ 722.565975] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 723.707632] Key type lgssc unregistered [ 723.849663] LNet: 46122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.852678] LNetError: 46122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.862358] LNet: Removed LNI 192.168.204.131@tcp [ 724.202727] Key type .llcrypt unregistered [ 724.203950] Key type ._llcrypt unregistered [ 734.678839] Key type ._llcrypt registered [ 734.680525] Key type .llcrypt registered [ 734.731198] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 735.147307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.154882] alg: No test for adler32 (adler32-zlib) [ 736.017982] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 736.107811] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 737.695176] Key type lgssc registered [ 738.148124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 740.964970] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 740.974403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.447263] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 761.567171] LustreError: 47466:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 788.192680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 790.361807] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 792.295246] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 958.246626] Lustre: server umount lustre-OST0000 complete [ 963.524795] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 964.518341] Key type lgssc unregistered [ 964.645435] LNet: 49009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.647992] LNetError: 49009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.656276] LNet: Removed LNI 192.168.204.131@tcp [ 964.925866] Key type .llcrypt unregistered [ 964.927073] Key type ._llcrypt unregistered [ 973.493300] Key type ._llcrypt registered [ 973.494584] Key type .llcrypt registered [ 973.535237] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 973.871316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.920648] alg: No test for adler32 (adler32-zlib) [ 974.774353] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 974.870632] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 976.455142] Key type lgssc registered [ 976.856649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 979.622550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 979.630486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.785598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.136546] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 984.994974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.105728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 986.377894] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 988.182308] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 989.449233] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 991.659896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.774385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.635823] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 995.513449] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 999.905138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.905992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.912198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1002.448226] Lustre: server umount lustre-OST0000 complete [ 1005.024508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.025177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1005.028451] Lustre: Skipped 1 previous similar message [ 1010.144155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1010.147167] Lustre: Skipped 2 previous similar messages [ 1015.264044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1015.266155] Lustre: Skipped 1 previous similar message [ 1018.335109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1018.393947] Lustre: server umount lustre-MDT0000 complete [ 1019.688055] LustreError: 50362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777593965 with bad export cookie 13879869267938606609 [ 1019.690104] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.692250] LustreError: 50362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1019.815220] Lustre: server umount lustre-MDT0001 complete [ 1023.547855] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1024.542821] Key type lgssc unregistered [ 1024.665350] LNet: 53559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.668946] LNetError: 53559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.677257] LNet: Removed LNI 192.168.204.131@tcp [ 1024.974322] Key type .llcrypt unregistered [ 1024.975353] Key type ._llcrypt unregistered [ 1033.749982] Key type ._llcrypt registered [ 1033.751431] Key type .llcrypt registered [ 1033.788129] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1034.085558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1034.130396] alg: No test for adler32 (adler32-zlib) [ 1034.979950] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1035.062426] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1036.639173] Key type lgssc registered [ 1037.016099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.762168] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1039.770933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1055.199222] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1060.319160] LustreError: 54906:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 1086.943151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.758765] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1090.552317] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.722016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.882047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.905920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1094.202119] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1097.011922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.145263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.445862] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1100.136210] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.266325] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1102.306128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1102.308887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.313178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.315366] Lustre: Skipped 1 previous similar message [ 1103.946961] Lustre: server umount lustre-OST0000 complete [ 1121.952863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.953274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.957095] Lustre: Skipped 2 previous similar messages [ 1124.059146] Lustre: server umount lustre-MDT0000 complete [ 1125.199226] LustreError: 54914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594070 with bad export cookie 16533104690423886064 [ 1125.202337] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1125.204108] LustreError: 54914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1131.366700] Lustre: server umount lustre-MDT0001 complete [ 1134.581578] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1135.554941] Key type lgssc unregistered [ 1135.682362] LNet: 58285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.684516] LNetError: 58285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.693323] LNet: Removed LNI 192.168.204.131@tcp [ 1135.991794] Key type .llcrypt unregistered [ 1135.992853] Key type ._llcrypt unregistered [ 1144.216207] Key type ._llcrypt registered [ 1144.217594] Key type .llcrypt registered [ 1144.253880] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1144.560054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.587060] alg: No test for adler32 (adler32-zlib) [ 1145.457552] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1145.536844] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1147.127158] Key type lgssc registered [ 1147.455193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.961128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.967144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.121162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1152.367843] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1155.000924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.122971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.362944] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1158.014170] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1159.126416] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1161.199490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.317292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1163.057891] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1164.878994] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1175.652583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1186.272571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1186.272994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1186.274436] LustreError: Skipped 1 previous similar message [ 1186.275081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1186.277697] Lustre: Skipped 1 previous similar message [ 1187.851131] Lustre: server umount lustre-OST0000 complete [ 1190.879795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.882357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.887055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.889333] Lustre: Skipped 1 previous similar message [ 1195.089768] Lustre: server umount lustre-MDT0000 complete [ 1196.193857] LustreError: 59637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594141 with bad export cookie 13690544939551080595 [ 1196.196533] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1196.197020] LustreError: 59637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1196.298693] Lustre: server umount lustre-MDT0001 complete [ 1199.430819] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1200.349468] Key type lgssc unregistered [ 1200.468300] LNet: 62850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.470521] LNetError: 62850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.481259] LNet: Removed LNI 192.168.204.131@tcp [ 1200.754050] Key type .llcrypt unregistered [ 1200.754977] Key type ._llcrypt unregistered [ 1209.666347] Key type ._llcrypt registered [ 1209.667478] Key type .llcrypt registered [ 1209.707203] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1210.066753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.121065] alg: No test for adler32 (adler32-zlib) [ 1210.957294] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1211.039745] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1212.615153] Key type lgssc registered [ 1212.967661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.562597] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:09:19 (1777594159) [ 1217.514889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1217.520234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.667269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.866772] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1222.575030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.684967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.875682] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1225.481034] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.712056] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1228.822737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1228.923767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.627022] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1232.310549] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1244.325982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1250.408932] Lustre: Failing over lustre-MDT0000 [ 1250.591783] Lustre: server umount lustre-MDT0000 complete [ 1253.343779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1253.347185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.348647] LustreError: 64508:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.358138] LustreError: 64508:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1253.661505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1253.704556] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1253.759124] Lustre: *** cfs_fail_loc=135, val=0*** [ 1253.795539] LustreError: 67278:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1253.797651] LustreError: 67278:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -2 [ 1253.800143] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1253.800143] [ 1253.802764] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1253.809032] LustreError: 67266:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1253.812277] LustreError: 67266:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1253.816111] LustreError: 67266:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1253.818604] LustreError: 67266:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1253.829754] Lustre: server umount lustre-MDT0000 complete [ 1253.831421] LustreError: 67266:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1254.882453] LustreError: 64513:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.882929] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.888524] LustreError: 64513:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1254.893436] Lustre: Skipped 1 previous similar message [ 1255.992548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.033794] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.112505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.127497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1257.371205] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1258.459424] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1261.539085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1261.546561] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1261.562067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1264.096128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.099698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.104829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.656582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1266.659659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.659660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.623222] Lustre: server umount lustre-OST0000 complete [ 1271.776399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1271.776813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.780502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.782646] Lustre: Skipped 1 previous similar message [ 1271.784201] Lustre: Skipped 1 previous similar message [ 1275.983546] Lustre: server umount lustre-MDT0000 complete [ 1276.895936] LustreError: 64512:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.900963] LustreError: 64512:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1277.155143] LustreError: 64491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594222 with bad export cookie 11098289286379383075 [ 1277.157298] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.162072] LustreError: 64491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1277.276800] Lustre: server umount lustre-MDT0001 complete [ 1280.512642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1281.425995] Key type lgssc unregistered [ 1281.543279] LNet: 69020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.545837] LNetError: 69020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.554230] LNet: Removed LNI 192.168.204.131@tcp [ 1281.826368] Key type .llcrypt unregistered [ 1281.827268] Key type ._llcrypt unregistered [ 1290.028949] Key type ._llcrypt registered [ 1290.029954] Key type .llcrypt registered [ 1290.061737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1290.434680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1290.454183] alg: No test for adler32 (adler32-zlib) [ 1291.298300] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1291.373737] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1292.951158] Key type lgssc registered [ 1293.281268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1295.738321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1295.745115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.888458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1298.088728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1300.742947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.855766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1302.062488] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1303.721986] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1304.794435] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1306.928574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1307.031104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.069025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1308.837784] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1310.460723] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1313.541431] Lustre: Failing over lustre-MDT0000 [ 1313.758225] Lustre: server umount lustre-MDT0000 complete [ 1316.321712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1316.322772] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.323796] LustreError: 71104:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.323804] LustreError: 71104:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1316.338948] Lustre: Skipped 2 previous similar messages [ 1316.815329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1316.836860] LustreError: 70389:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1316.846846] LustreError: 70389:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1316.867381] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1316.871736] Lustre: 70386:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 1316.930109] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1316.980447] LustreError: 73139:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1316.983722] LustreError: 73139:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -9 [ 1316.987923] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1316.987923] [ 1316.992897] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1316.999414] LustreError: 73126:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1317.002817] LustreError: 73126:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1317.005683] LustreError: 73126:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1317.007915] LustreError: 73126:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1317.018371] Lustre: server umount lustre-MDT0000 complete [ 1317.019890] LustreError: 73126:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1319.142376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.186526] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.279349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.301438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1320.579272] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1321.948945] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1324.515896] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1324.524780] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1324.541261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1326.559754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.563186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.567069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.633713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1329.634825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1329.637431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.538699] Lustre: server umount lustre-OST0000 complete [ 1334.752504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1334.752674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1334.758845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.763322] Lustre: Skipped 2 previous similar messages [ 1338.958668] Lustre: server umount lustre-MDT0000 complete [ 1339.872176] LustreError: 70393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1339.879133] LustreError: 70393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1340.205093] LustreError: 71112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594285 with bad export cookie 17016423460846988067 [ 1340.208509] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.209160] LustreError: 71112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1340.320676] Lustre: server umount lustre-MDT0001 complete [ 1343.816859] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1344.847446] Key type lgssc unregistered [ 1344.968295] LNet: 74881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.970716] LNetError: 74881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.982690] LNet: Removed LNI 192.168.204.131@tcp [ 1345.284825] Key type .llcrypt unregistered [ 1345.286036] Key type ._llcrypt unregistered [ 1349.296480] Key type ._llcrypt registered [ 1349.297827] Key type .llcrypt registered [ 1349.334907] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1349.434403] Key type .llcrypt unregistered [ 1349.435773] Key type ._llcrypt unregistered [ 1358.010414] Key type ._llcrypt registered [ 1358.011746] Key type .llcrypt registered [ 1358.047590] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1358.347170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1358.352931] alg: No test for adler32 (adler32-zlib) [ 1359.199871] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1359.288257] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1360.871141] Key type lgssc registered [ 1361.230354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1363.825738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1363.838127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.998704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.251564] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1368.882347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.993244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.189450] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1371.845784] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.941357] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1375.063140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1375.168459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.230516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1376.929827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1378.492258] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1384.418318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1384.420055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.421393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1384.423912] Lustre: Skipped 1 previous similar message [ 1384.426439] Lustre: Skipped 1 previous similar message [ 1388.050792] Lustre: server umount lustre-OST0000 complete [ 1389.536233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1389.536317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.536814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1389.544481] Lustre: Skipped 1 previous similar message [ 1394.656173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.659108] Lustre: Skipped 2 previous similar messages [ 1395.282920] Lustre: server umount lustre-MDT0000 complete [ 1396.500521] LustreError: 76703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594341 with bad export cookie 8581673312144396661 [ 1396.503798] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1396.504684] LustreError: 76703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1396.616060] Lustre: server umount lustre-MDT0001 complete [ 1400.263851] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1401.257986] Key type lgssc unregistered [ 1401.381340] LNet: 79896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.383886] LNetError: 79896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.392334] LNet: Removed LNI 192.168.204.131@tcp [ 1401.658838] Key type .llcrypt unregistered [ 1401.660192] Key type ._llcrypt unregistered [ 1409.815281] Key type ._llcrypt registered [ 1409.816471] Key type .llcrypt registered [ 1409.849094] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1410.226841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.239207] alg: No test for adler32 (adler32-zlib) [ 1411.086386] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1411.166831] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1412.743129] Key type lgssc registered [ 1413.065365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.455403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.460527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.603197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.818271] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1420.436911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.551749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.854266] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1423.634412] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.711172] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.729472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.830795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.581583] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1430.162350] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1441.188766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1450.463868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1450.465959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.471132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.489110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1451.489553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.495802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1454.152732] Lustre: server umount lustre-OST0000 complete [ 1456.608810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.609134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.609390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.609395] Lustre: Skipped 1 previous similar message [ 1456.612460] Lustre: Skipped 1 previous similar message [ 1461.455702] Lustre: server umount lustre-MDT0000 complete [ 1461.728138] LustreError: 81991:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.734555] LustreError: 81991:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1462.601020] LustreError: 81251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594407 with bad export cookie 14581193061140607290 [ 1462.603312] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.604276] LustreError: 81251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1462.705063] Lustre: server umount lustre-MDT0001 complete [ 1465.809507] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1466.778053] Key type lgssc unregistered [ 1466.904720] LNet: 84474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.906880] LNetError: 84474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.918329] LNet: Removed LNI 192.168.204.131@tcp [ 1467.194253] Key type .llcrypt unregistered [ 1467.195843] Key type ._llcrypt unregistered [ 1474.947067] Key type ._llcrypt registered [ 1474.947916] Key type .llcrypt registered [ 1474.980495] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1475.261169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.273648] alg: No test for adler32 (adler32-zlib) [ 1476.114074] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1476.182370] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1477.759135] Key type lgssc registered [ 1478.094506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1480.496490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1480.501652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.640424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.787608] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1485.325896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1485.457193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1486.648023] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1488.222240] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.241293] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1491.232792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.337748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1492.388624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 1493.020776] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1494.702728] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1500.640642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1500.641929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.643271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1500.646130] Lustre: Skipped 1 previous similar message [ 1504.658086] Lustre: server umount lustre-OST0000 complete [ 1506.271923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1506.273830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.276879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.278287] Lustre: Skipped 1 previous similar message [ 1510.880165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1510.880399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.023674] Lustre: server umount lustre-MDT0000 complete [ 1513.187093] LustreError: 85834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594458 with bad export cookie 8032140537456359938 [ 1513.190071] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.191659] LustreError: 85834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1513.303222] Lustre: server umount lustre-MDT0001 complete [ 1516.765225] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1517.726677] Key type lgssc unregistered [ 1517.840336] LNet: 89023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.842646] LNetError: 89023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.851277] LNet: Removed LNI 192.168.204.131@tcp [ 1518.101616] Key type .llcrypt unregistered [ 1518.102572] Key type ._llcrypt unregistered [ 1526.252641] Key type ._llcrypt registered [ 1526.254255] Key type .llcrypt registered [ 1526.290423] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1526.670323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.680506] alg: No test for adler32 (adler32-zlib) [ 1527.530440] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1527.623392] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1529.207164] Key type lgssc registered [ 1529.579471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1532.084406] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1532.091369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1547.551223] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1552.671154] LustreError: 90369:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 1579.295197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.137336] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1582.884311] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1584.646438] Lustre: server umount lustre-OST0000 complete [ 1587.127252] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:15:31 (1777594531) [ 1589.147969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.291516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.513975] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1593.066361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.171450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.298438] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1595.856284] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1596.937106] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1599.039521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.145774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.166341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 1600.891807] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1602.512622] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1608.674026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.675861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1608.677276] Lustre: Skipped 1 previous similar message [ 1608.677908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1608.679780] Lustre: Skipped 1 previous similar message [ 1612.681505] Lustre: server umount lustre-OST0000 complete [ 1618.912751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.912801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1618.913676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.916354] Lustre: Skipped 1 previous similar message [ 1624.032088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.034113] Lustre: Skipped 2 previous similar messages [ 1628.127098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1628.177568] Lustre: server umount lustre-MDT0000 complete [ 1629.152035] LustreError: 91890:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1629.157043] LustreError: 91890:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1629.319065] LustreError: 91872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594574 with bad export cookie 12955126481141184268 [ 1629.321385] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.321994] LustreError: 91872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1629.420279] Lustre: server umount lustre-MDT0001 complete [ 1632.536685] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1633.437418] Key type lgssc unregistered [ 1633.555350] LNet: 95219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.557506] LNetError: 95219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.564235] LNet: Removed LNI 192.168.204.131@tcp [ 1633.822382] Key type .llcrypt unregistered [ 1633.823404] Key type ._llcrypt unregistered [ 1647.535218] Key type ._llcrypt registered [ 1647.536314] Key type .llcrypt registered [ 1647.576254] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1647.939444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1647.976538] alg: No test for adler32 (adler32-zlib) [ 1648.834846] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1648.924961] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1650.511191] Key type lgssc registered [ 1650.925605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1653.511406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1653.518251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.673222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.901184] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1658.746028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.881586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1660.258930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1662.102622] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1663.293598] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1665.656879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1665.766877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.470751] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1669.027989] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1680.100933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 1689.568405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1689.571760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.575873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.593181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.594440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.594931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.135661] Lustre: server umount lustre-OST0000 complete [ 1694.687905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1694.690744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.695750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1694.698340] Lustre: Skipped 1 previous similar message [ 1700.499177] Lustre: server umount lustre-MDT0000 complete [ 1700.832943] LustreError: 97363:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.837845] LustreError: 97363:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1701.692983] LustreError: 98078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594647 with bad export cookie 698151377387826593 [ 1701.695824] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.696069] LustreError: 98078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1701.823048] Lustre: server umount lustre-MDT0001 complete [ 1705.162323] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1706.170222] Key type lgssc unregistered [ 1706.304391] LNet: 100556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.307036] LNetError: 100556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.316363] LNet: Removed LNI 192.168.204.131@tcp [ 1706.609103] Key type .llcrypt unregistered [ 1706.610803] Key type ._llcrypt unregistered [ 1714.022135] Key type ._llcrypt registered [ 1714.023611] Key type .llcrypt registered [ 1714.066243] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1714.601307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.611791] alg: No test for adler32 (adler32-zlib) [ 1715.460652] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1715.551475] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1717.135108] Key type lgssc registered [ 1717.513448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.910560] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.915411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.327230] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1740.447196] LustreError: 101760:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 1767.071166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1768.928383] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1770.664231] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1772.863768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.957644] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1772.962081] LustreError: 102669:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1772.964829] LustreError: 102669:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1772.968044] LustreError: 102669:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1772.996145] Lustre: server umount lustre-MDT0000 complete [ 1772.997516] LustreError: 102669:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1784.838387] Lustre: server umount lustre-OST0000 complete [ 1787.883462] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1790.927566] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1802.538649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.614911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1802.625931] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1802.661146] Lustre: lustre-MDT0000: new disk, initializing [ 1802.683869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.689933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1803.788550] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1807.781190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1807.823760] Lustre: 105422:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1807.847992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1807.851033] Lustre: Skipped 1 previous similar message [ 1807.886105] Lustre: lustre-MDT0001: new disk, initializing [ 1807.920603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.932071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1807.936881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1809.182995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1812.189963] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1813.371189] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1815.617859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1815.727909] Lustre: lustre-OST0000: new disk, initializing [ 1815.729730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1815.760711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.656561] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1817.847599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1817.850520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1817.885185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1820.731765] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1822.199671] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1822.283573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1823.726807] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1823.809076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1828.322220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1828.322554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1828.324085] LustreError: Skipped 1 previous similar message [ 1828.327349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1828.330252] Lustre: Skipped 1 previous similar message [ 1830.536373] Lustre: server umount lustre-OST0000 complete [ 1833.442916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.443564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.447169] Lustre: Skipped 2 previous similar messages [ 1833.451917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1838.559894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.561723] Lustre: Skipped 2 previous similar messages [ 1843.680055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1843.682107] Lustre: Skipped 1 previous similar message [ 1846.239099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1846.312058] Lustre: server umount lustre-MDT0000 complete [ 1847.592280] LustreError: 105413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594792 with bad export cookie 1876255934707322705 [ 1847.595055] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.596130] LustreError: 105413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1847.716174] Lustre: server umount lustre-MDT0001 complete [ 1850.370160] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:19:54 (1777594794) [ 1851.414441] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1856.571325] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1859.578070] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1870.803416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.889864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1870.900355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1870.937197] Lustre: lustre-MDT0000: new disk, initializing [ 1870.959224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.965887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1872.273396] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1876.321934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.359888] Lustre: 111871:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1876.390585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1876.392285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1876.394157] Lustre: Skipped 1 previous similar message [ 1876.428473] Lustre: lustre-MDT0001: new disk, initializing [ 1876.486205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1876.489801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1877.845052] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1880.932568] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1882.093539] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1884.346213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1884.460658] Lustre: lustre-OST0000: new disk, initializing [ 1884.463952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1884.499516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.501839] Lustre: Skipped 1 previous similar message [ 1886.447084] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1889.445955] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1890.777442] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1890.809278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1890.812349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1890.842271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1891.870316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 1893.149308] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1893.220477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1895.906455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1895.906885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.910098] LustreError: Skipped 1 previous similar message [ 1895.914298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1895.917887] Lustre: Skipped 1 previous similar message [ 1899.920406] Lustre: server umount lustre-OST0000 complete [ 1902.048178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.052068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.058331] Lustre: Skipped 1 previous similar message [ 1906.145056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1915.359104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1915.414527] Lustre: server umount lustre-MDT0000 complete [ 1916.604148] LustreError: 112646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594861 with bad export cookie 1876255934707323573 [ 1916.608570] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.610718] LustreError: 112646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.716630] Lustre: server umount lustre-MDT0001 complete [ 1919.455148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.631035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1920.793590] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1923.490228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.884484] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1926.600630] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1927.751703] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1929.952240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1931.853943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1933.564193] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1951.541103] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1953.759787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.762599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.767136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.769076] Lustre: Skipped 5 previous similar messages [ 1958.601516] Lustre: server umount lustre-OST0000 complete [ 1974.239085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1974.299679] Lustre: server umount lustre-MDT0000 complete [ 1975.522127] LustreError: 117954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594920 with bad export cookie 1876255934707324455 [ 1975.525234] LustreError: 117954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1975.525567] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1975.639875] Lustre: server umount lustre-MDT0001 complete [ 1979.257565] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1980.213899] Key type lgssc unregistered [ 1980.336313] LNet: 118979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.339229] LNetError: 118979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.347301] LNet: Removed LNI 192.168.204.131@tcp [ 1980.618887] Key type .llcrypt unregistered [ 1980.619927] Key type ._llcrypt unregistered [ 1986.318282] Key type ._llcrypt registered [ 1986.320207] Key type .llcrypt registered [ 1986.361838] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1992.498728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1992.798978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1992.806690] alg: No test for adler32 (adler32-zlib) [ 1993.657792] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 1993.739208] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1995.319131] Key type lgssc registered [ 1995.686599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2006.553103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2006.558307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.647755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2007.661751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2007.700450] Lustre: lustre-MDT0000: new disk, initializing [ 2007.725506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2007.733807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2009.023727] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2013.062600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.105503] Lustre: 122071:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2013.127460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2013.130186] Lustre: Skipped 1 previous similar message [ 2013.166712] Lustre: lustre-MDT0001: new disk, initializing [ 2013.201039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.213154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2013.218850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2014.512237] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2017.544916] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2018.708582] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.895483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.991482] Lustre: lustre-OST0000: new disk, initializing [ 2020.993341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2021.023158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.840367] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2025.890375] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2027.273457] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2029.045577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2029.048752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2029.079388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2029.382340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2030.637502] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2030.703941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2034.146196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.146536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.148707] LustreError: Skipped 1 previous similar message [ 2034.149470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2034.154287] Lustre: Skipped 1 previous similar message [ 2037.456297] Lustre: server umount lustre-OST0000 complete [ 2038.751755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2038.754371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.760268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.763016] Lustre: Skipped 1 previous similar message [ 2044.384148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2044.386016] Lustre: Skipped 3 previous similar messages [ 2044.828157] Lustre: server umount lustre-MDT0000 complete [ 2046.055351] LustreError: 122062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777594991 with bad export cookie 7048832903341671319 [ 2046.058785] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.058821] LustreError: 122062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.169384] Lustre: server umount lustre-MDT0001 complete [ 2048.789195] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:23:13 (1777594993) [ 2050.973267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.178992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.476901] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2055.211207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.330535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.647680] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2058.372821] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.560606] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2060.203585] LustreError: 126114:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2060.205522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.206641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.207270] LustreError: 126114:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2060.211419] Lustre: Skipped 2 previous similar messages [ 2060.213318] LustreError: 126114:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2065.376238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.378613] Lustre: Skipped 2 previous similar messages [ 2066.384221] Lustre: server umount lustre-MDT0000 complete [ 2067.560030] LustreError: 126066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595012 with bad export cookie 7048832903341672187 [ 2067.562819] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2067.563159] LustreError: 126066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2067.674766] Lustre: server umount lustre-MDT0001 complete [ 2070.249990] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:23:34 (1777595014) [ 2072.268688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.711461] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2092.831443] LustreError: 128268:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2118.431258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2120.148450] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2121.851389] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2122.570598] Lustre: server umount lustre-OST0000 complete [ 2125.318771] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:24:29 (1777595069) [ 2127.530476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.715968] LustreError: 129598:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.725543] LustreError: 129598:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2129.340068] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2132.266751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.627660] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2135.330248] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2136.569400] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2139.094498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2139.225406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.228872] Lustre: Skipped 2 previous similar messages [ 2141.184032] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2142.847000] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2163.167986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2163.171094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.176210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2166.033894] Lustre: server umount lustre-MDT0000 complete [ 2167.204068] LustreError: 130315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595112 with bad export cookie 7048832903341672831 [ 2167.207417] LustreError: 130315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.207796] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.311271] Lustre: server umount lustre-MDT0001 complete [ 2177.685472] Lustre: server umount lustre-OST0000 complete [ 2180.235687] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:25:24 (1777595124) [ 2182.368461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.515863] LustreError: 132930:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.538512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.799026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2186.457094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.750739] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2189.395040] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2190.473573] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.543215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2194.492742] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2196.150159] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2197.426542] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2207.014904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2207.680399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2208.934657] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2209.016334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2211.295718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2211.298108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.301755] Lustre: Skipped 2 previous similar messages [ 2211.303450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2211.305397] Lustre: Skipped 3 previous similar messages [ 2215.758562] Lustre: server umount lustre-OST0000 complete [ 2217.441411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2231.263125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.367096] Lustre: server umount lustre-MDT0000 complete [ 2232.804308] LustreError: 132949:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.815447] LustreError: 132949:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2233.268605] LustreError: 132911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595178 with bad export cookie 7048832903341673888 [ 2233.271722] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.276724] LustreError: 132911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2233.431565] Lustre: server umount lustre-MDT0001 complete [ 2237.489996] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:26:21 (1777595181) [ 2241.239622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.735652] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2261.855395] LustreError: 136781:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2287.455468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.458897] Lustre: Skipped 2 previous similar messages [ 2289.688726] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2291.827268] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.268874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.419354] LustreError: 136806:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2294.463819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2295.751906] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2298.896209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.468920] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2302.353688] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2303.602229] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2305.218409] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2305.318147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2306.947446] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2307.037850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2309.600738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2309.603955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.610422] Lustre: Skipped 3 previous similar messages [ 2309.613691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2309.618688] Lustre: Skipped 8 previous similar messages [ 2321.887234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2321.969943] Lustre: server umount lustre-MDT0000 complete [ 2322.912070] LustreError: 138824:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.920495] LustreError: 138824:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2323.456205] LustreError: 136787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595268 with bad export cookie 7048832903341674896 [ 2323.459548] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.461408] LustreError: 136787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2323.597921] Lustre: server umount lustre-MDT0001 complete [ 2335.457697] Lustre: server umount lustre-OST0000 complete [ 2338.637593] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:28:02 (1777595282) [ 2341.580195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.023537] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2362.143456] LustreError: 140597:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2390.129979] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2392.449300] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2395.294336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.499927] LustreError: 140622:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.560258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2397.107490] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2400.935523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.908431] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2405.272744] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2406.793351] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2409.747067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2409.828479] Lustre: lustre-OST0001: new disk, initializing [ 2409.830462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2412.120860] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2415.472246] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2417.066028] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2417.146533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2417.150936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2417.186421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2418.182805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2419.645141] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2419.725313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2421.215892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.221132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.226184] Lustre: Skipped 2 previous similar messages [ 2421.229059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2421.231664] Lustre: Skipped 6 previous similar messages [ 2426.494315] Lustre: server umount lustre-OST0000 complete [ 2427.363870] LustreError: 140613:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.371269] LustreError: 140613:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2442.207122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2442.270148] Lustre: server umount lustre-OST0001 complete [ 2449.749877] Lustre: server umount lustre-MDT0000 complete [ 2450.906142] LustreError: 140605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595396 with bad export cookie 7048832903341675848 [ 2450.909294] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.911127] LustreError: 140605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2451.028258] Lustre: server umount lustre-MDT0001 complete [ 2458.147196] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:30:02 (1777595402) [ 2458.605930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2459.058366] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2459.548270] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:30:03 (1777595403) [ 2461.550598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.640254] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2461.650784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2461.719421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.722294] Lustre: Skipped 6 previous similar messages [ 2462.909559] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2465.535352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.569336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2465.577641] Lustre: 146546:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2466.871782] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2468.512247] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2469.607763] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.682179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.743967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2473.556924] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2475.139833] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2476.431054] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2477.062735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2477.517941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2478.757656] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2478.821166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2482.144798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2482.144994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.147217] LustreError: Skipped 5 previous similar messages [ 2482.152870] Lustre: Skipped 6 previous similar messages [ 2485.510494] Lustre: server umount lustre-OST0000 complete [ 2499.007166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2500.903564] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2502.179607] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.183043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.183081] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2502.189404] Lustre: Skipped 1 previous similar message [ 2502.534649] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2503.913962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2503.986660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2505.200402] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2505.266650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2506.362636] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff96af03fc9000.ost_server_uuid 50 [ 2506.817726] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff96af03fc9000.ost_server_uuid in FULL state after 0 sec [ 2506.870381] LustreError: 146553:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.31@tcp) returned error from blocking AST (req@000000008b02d91b x1863943390948992 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000008a1e4f7b/0x61d27c2de4c41e94 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.31@tcp remote: 0x7b54419394344cbd expref: 5 pid: 146552 timeout: 2606 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2506.881478] LustreError: lustre-MDT0000: A client on nid 192.168.204.31@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2506.883987] LustreError: 146541:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.31@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000008a1e4f7b/0x61d27c2de4c41e94 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.31@tcp remote: 0x7b54419394344cbd expref: 6 pid: 146552 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2513.863192] Lustre: server umount lustre-OST0000 complete [ 2521.181321] Lustre: server umount lustre-MDT0000 complete [ 2522.346274] LustreError: 147272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595467 with bad export cookie 7048832903341677031 [ 2522.348802] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.350623] LustreError: 147272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2528.502623] Lustre: server umount lustre-MDT0001 complete [ 2531.718747] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2532.741147] Key type lgssc unregistered [ 2532.876332] LNet: 152021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.878541] LNetError: 152021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.887281] LNet: Removed LNI 192.168.204.131@tcp [ 2533.178663] Key type .llcrypt unregistered [ 2533.179913] Key type ._llcrypt unregistered [ 2541.593989] Key type ._llcrypt registered [ 2541.595211] Key type .llcrypt registered [ 2541.632140] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2541.999541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2542.011274] alg: No test for adler32 (adler32-zlib) [ 2542.857545] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2542.934378] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2544.511174] Key type lgssc registered [ 2544.863513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2547.345457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2547.351831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.499764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.756523] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2552.396635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.514294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2553.769190] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2555.473737] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2556.599435] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2558.670359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.776976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.503696] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2562.105885] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2573.094048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2580.017052] Lustre: Failing over lustre-MDT0000 [ 2580.157038] Lustre: server umount lustre-MDT0000 complete [ 2583.520654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.521346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.522143] LustreError: 153394:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.522149] LustreError: 153394:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2583.523865] Lustre: Skipped 1 previous similar message [ 2588.639964] LustreError: 153393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.644185] LustreError: 153393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 2591.007465] LustreError: 155007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.31@tcp arrived at 1777595536 with bad export cookie 6225643561699392126 [ 2591.012692] LustreError: 155007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2593.173336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.207819] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.289708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.312246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2594.459703] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2595.124829] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2595.127227] Lustre: lustre-MDT0000: Denying connection for new client c519b04b-790f-4805-84e5-12e2164bc413 (at 192.168.204.31@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2598.371739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2602.975887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.978327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.982053] Lustre: Skipped 1 previous similar message [ 2602.984091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2603.488558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.608958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.610972] Lustre: Skipped 2 previous similar messages [ 2613.729280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2613.730955] Lustre: Skipped 1 previous similar message [ 2616.287104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2616.343776] Lustre: server umount lustre-OST0000 complete [ 2617.525563] LustreError: 156785:0:(ldlm_lib.c:2986:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2617.528234] Lustre: 156149:0:(ldlm_lib.c:2389:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2617.531063] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2617.537368] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2617.541793] LustreError: 156149:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000081c2b98d x1863943966463232/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2617.547837] LustreError: 156149:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2617.551347] LustreError: 156149:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2617.554769] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 2617.652240] Lustre: server umount lustre-MDT0000 complete [ 2618.849404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.853907] Lustre: Skipped 1 previous similar message [ 2618.856239] LustreError: 154800:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.856774] LustreError: 153373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595564 with bad export cookie 6225643561699392476 [ 2618.859830] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.860498] LustreError: 152704:0:(import.c:706:ptlrpc_connect_import_locked()) can't connect to a closed import [ 2618.869989] LustreError: 153373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2625.011841] Lustre: server umount lustre-MDT0001 complete [ 2628.427850] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2629.486948] Key type lgssc unregistered [ 2629.611351] LNet: 157518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2629.613679] LNetError: 157518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2629.624315] LNet: Removed LNI 192.168.204.131@tcp [ 2629.917178] Key type .llcrypt unregistered [ 2629.918475] Key type ._llcrypt unregistered [ 2638.211257] Key type ._llcrypt registered [ 2638.212679] Key type .llcrypt registered [ 2638.248558] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2638.600597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.608953] alg: No test for adler32 (adler32-zlib) [ 2639.446429] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2639.515073] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2641.087150] Key type lgssc registered [ 2641.452746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2643.960083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2643.965818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.126781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.396350] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2649.160193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.277839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2650.599208] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2652.369254] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2653.524867] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2655.758955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.867437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.728420] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2659.521439] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.269803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2676.704806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.705322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.706837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2676.709681] Lustre: Skipped 1 previous similar message [ 2681.825286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2681.828703] Lustre: Skipped 2 previous similar messages [ 2682.446887] Lustre: server umount lustre-OST0000 complete [ 2685.407908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2685.411399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2685.417288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.944977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.819340] Lustre: server umount lustre-MDT0000 complete [ 2691.002953] LustreError: 158873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595636 with bad export cookie 5623838733201163325 [ 2691.006383] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2691.006678] LustreError: 158873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2691.126978] Lustre: server umount lustre-MDT0001 complete [ 2694.563303] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2695.593538] Key type lgssc unregistered [ 2695.731394] LNet: 162087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.734193] LNetError: 162087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.743417] LNet: Removed LNI 192.168.204.131@tcp [ 2696.041359] Key type .llcrypt unregistered [ 2696.042872] Key type ._llcrypt unregistered [ 2707.302908] Key type ._llcrypt registered [ 2707.304186] Key type .llcrypt registered [ 2707.346050] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2707.672529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2707.703078] alg: No test for adler32 (adler32-zlib) [ 2708.555194] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2708.636553] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2710.223124] Key type lgssc registered [ 2710.565108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2713.100270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2713.106523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.264686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.555779] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2718.297935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.418198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2719.660962] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2721.382751] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2722.515320] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2724.627043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.752516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2725.798238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2726.538972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2728.227340] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2730.976220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2730.978695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.982955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.024740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.024750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.027051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.526737] Lustre: server umount lustre-OST0000 complete [ 2738.144625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.148821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2738.150539] Lustre: Skipped 2 previous similar messages [ 2743.263939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2743.265934] Lustre: Skipped 1 previous similar message [ 2743.893248] Lustre: server umount lustre-MDT0000 complete [ 2745.108081] LustreError: 163728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595690 with bad export cookie 5430509920040997151 [ 2745.111194] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.112404] LustreError: 163728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.229101] Lustre: server umount lustre-MDT0001 complete [ 2748.578065] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2749.566903] Key type lgssc unregistered [ 2749.698417] LNet: 166920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2749.701977] LNetError: 166920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2749.708272] LNet: Removed LNI 192.168.204.131@tcp [ 2749.996936] Key type .llcrypt unregistered [ 2749.998552] Key type ._llcrypt unregistered [ 2758.194864] Key type ._llcrypt registered [ 2758.195994] Key type .llcrypt registered [ 2758.238080] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2758.573705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.628738] alg: No test for adler32 (adler32-zlib) [ 2759.490166] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2759.576616] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2761.159140] Key type lgssc registered [ 2761.534564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2765.141675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2765.149021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.287686] Lustre: *** cfs_fail_loc=135, val=0*** [ 2766.334064] LustreError: 168403:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2766.336606] LustreError: 168403:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -2 [ 2766.339744] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2766.339744] [ 2766.342904] LustreError: MGC192.168.204.131@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2766.347213] LustreError: 168365:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2766.349959] LustreError: 168365:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2766.352847] LustreError: 168365:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2766.356522] LustreError: 168365:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2766.393696] Lustre: server umount lustre-MDT0000 complete [ 2766.395166] LustreError: 168365:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2768.559147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.702568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2769.952562] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2772.714530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.841385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.144222] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2775.868941] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.041432] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2777.698629] LustreError: 168804:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2777.702207] LustreError: 168804:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2777.704993] LustreError: 168804:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2778.080449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2778.080918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.081616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2778.091807] Lustre: Skipped 1 previous similar message [ 2783.200676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2783.203042] Lustre: Skipped 2 previous similar messages [ 2783.829776] Lustre: server umount lustre-MDT0000 complete [ 2785.030155] LustreError: 168758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595730 with bad export cookie 15929191506437415947 [ 2785.032889] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2785.034520] LustreError: 168758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2785.146131] Lustre: server umount lustre-MDT0001 complete [ 2788.555734] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2789.657863] Key type lgssc unregistered [ 2789.788494] LNet: 170848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.792737] LNetError: 170848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.799331] LNet: Removed LNI 192.168.204.131@tcp [ 2790.129282] Key type .llcrypt unregistered [ 2790.130620] Key type ._llcrypt unregistered [ 2796.672075] Key type ._llcrypt registered [ 2796.673162] Key type .llcrypt registered [ 2796.706551] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2796.795370] Key type .llcrypt unregistered [ 2796.796421] Key type ._llcrypt unregistered [ 2803.289648] Key type ._llcrypt registered [ 2803.290674] Key type .llcrypt registered [ 2803.323963] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2803.588176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2803.625372] alg: No test for adler32 (adler32-zlib) [ 2804.472721] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2804.555366] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2806.135183] Key type lgssc registered [ 2806.503026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2809.002958] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2809.008535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.416480] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2829.535199] LustreError: 172920:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2856.159272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.975784] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2859.651796] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2861.815136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.982864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.013965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2863.238944] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2865.940727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.073826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2867.354769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2869.063477] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2870.223834] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.157044] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2876.386983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.387530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.390552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.395876] Lustre: Skipped 1 previous similar message [ 2880.719399] Lustre: server umount lustre-OST0000 complete [ 2886.624951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.625019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2886.625720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2886.625726] Lustre: Skipped 1 previous similar message [ 2886.629331] Lustre: Skipped 1 previous similar message [ 2891.743966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.746223] Lustre: Skipped 2 previous similar messages [ 2895.840293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.842462] Lustre: Skipped 1 previous similar message [ 2896.351125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2896.413283] Lustre: server umount lustre-MDT0000 complete [ 2897.605049] LustreError: 172927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595842 with bad export cookie 7990144547485403786 [ 2897.608266] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.610117] LustreError: 172927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2897.732120] Lustre: server umount lustre-MDT0001 complete [ 2901.120653] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2902.109323] Key type lgssc unregistered [ 2902.233320] LNet: 176426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.235542] LNetError: 176426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.246306] LNet: Removed LNI 192.168.204.131@tcp [ 2902.543572] Key type .llcrypt unregistered [ 2902.544792] Key type ._llcrypt unregistered [ 2910.993975] Key type ._llcrypt registered [ 2910.995451] Key type .llcrypt registered [ 2911.033110] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2911.349502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2911.356375] alg: No test for adler32 (adler32-zlib) [ 2912.203851] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2912.285834] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2913.863155] Key type lgssc registered [ 2914.213694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2916.761872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2916.767618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.930537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.231441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2922.038377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.166529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2923.468532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2925.270896] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2926.449343] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2928.570658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2928.610636] LustreError: 179392:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.617201] LustreError: 179392:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2928.682645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.425770] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2931.751420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 2932.075898] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.166291] Lustre: Failing over lustre-MDT0000 [ 2934.254117] Lustre: server umount lustre-MDT0000 complete [ 2936.801475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.802986] LustreError: 177795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.805152] Lustre: Skipped 1 previous similar message [ 2936.813637] LustreError: 177795:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2938.338083] LustreError: 177795:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.920153] LustreError: 178508:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2941.926171] LustreError: 178508:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2947.040234] LustreError: 179954:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.046481] LustreError: 179954:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2947.130970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.177763] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2947.183289] Lustre: 177790:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.204.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2947.275490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.299530] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2948.571944] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2948.586283] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2950.114277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 2952.676851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2952.683532] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2952.702170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 2957.178243] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 2962.400068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.403949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.410702] Lustre: Skipped 1 previous similar message [ 2962.413259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.914329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2962.916199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.916827] LustreError: Skipped 1 previous similar message [ 2965.323282] Lustre: server umount lustre-OST0000 complete [ 2968.032993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.033457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.033691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2968.039068] Lustre: Skipped 2 previous similar messages [ 2972.695902] Lustre: server umount lustre-MDT0000 complete [ 2973.153101] LustreError: 177798:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2973.158129] LustreError: 177798:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2973.884204] LustreError: 177779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777595919 with bad export cookie 10596969751705232054 [ 2973.888026] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.888168] LustreError: 177779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2974.008829] Lustre: server umount lustre-MDT0001 complete [ 2977.534263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2978.510568] Key type lgssc unregistered [ 2978.633304] LNet: 182402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.635875] LNetError: 182402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.643335] LNet: Removed LNI 192.168.204.131@tcp [ 2978.937782] Key type .llcrypt unregistered [ 2978.939128] Key type ._llcrypt unregistered [ 2987.609259] Key type ._llcrypt registered [ 2987.610230] Key type .llcrypt registered [ 2987.649984] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2987.964521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2987.970579] alg: No test for adler32 (adler32-zlib) [ 2988.818112] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 2988.896787] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2990.479149] Key type lgssc registered [ 2990.841395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2993.331075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2993.337577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.489610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.764328] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2998.474380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.598587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2999.839845] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3001.592974] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3002.719682] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3004.899988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3005.013245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.868139] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3008.659149] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3019.365315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3024.913856] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3027.335649] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3036.366538] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3044.319723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.321858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.325417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.345750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.346272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.346725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3049.551816] Lustre: server umount lustre-OST0000 complete [ 3055.071782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3055.075513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.081601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.083798] Lustre: Skipped 1 previous similar message [ 3060.704126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.706490] Lustre: Skipped 2 previous similar messages [ 3065.311088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3065.375126] Lustre: server umount lustre-MDT0000 complete [ 3065.823891] LustreError: 184482:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.829222] LustreError: 184482:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3066.545043] LustreError: 184490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596011 with bad export cookie 2393937774612907484 [ 3066.547573] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3066.548138] LustreError: 184490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3066.668507] Lustre: server umount lustre-MDT0001 complete [ 3069.965376] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3070.905798] Key type lgssc unregistered [ 3071.033323] LNet: 187133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3071.036821] LNetError: 187133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3071.049297] LNet: Removed LNI 192.168.204.131@tcp [ 3071.320730] Key type .llcrypt unregistered [ 3071.321801] Key type ._llcrypt unregistered [ 3079.764486] Key type ._llcrypt registered [ 3079.765698] Key type .llcrypt registered [ 3079.801979] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3080.157746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3080.228696] alg: No test for adler32 (adler32-zlib) [ 3081.079786] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3081.161974] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3082.743173] Key type lgssc registered [ 3083.102055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3085.608384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3085.614609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.767056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3088.013533] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3090.702898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.832601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3092.097931] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3093.819571] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3094.972881] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3097.094931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3097.207705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.002316] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3100.701329] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3112.550604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3117.874927] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3126.717061] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3135.816512] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3143.419038] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3152.863801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3152.867220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.873116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3153.889069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3153.890731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.894511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.790201] Lustre: server umount lustre-OST0000 complete [ 3157.471789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3157.473944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.477616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.479631] Lustre: Skipped 1 previous similar message [ 3164.127971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.130782] Lustre: Skipped 3 previous similar messages [ 3169.248206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3169.250421] Lustre: Skipped 1 previous similar message [ 3171.295124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3171.355807] Lustre: server umount lustre-MDT0000 complete [ 3172.544069] LustreError: 188486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596117 with bad export cookie 669159216553417882 [ 3172.547147] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.548707] LustreError: 188486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3172.654718] Lustre: server umount lustre-MDT0001 complete [ 3175.905683] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3176.880609] Key type lgssc unregistered [ 3177.001360] LNet: 193623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3177.004438] LNetError: 193623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3177.018276] LNet: Removed LNI 192.168.204.131@tcp [ 3177.295806] Key type .llcrypt unregistered [ 3177.296925] Key type ._llcrypt unregistered [ 3185.647093] Key type ._llcrypt registered [ 3185.648095] Key type .llcrypt registered [ 3185.682340] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3186.005261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3186.067507] alg: No test for adler32 (adler32-zlib) [ 3186.917188] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3187.000518] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3188.583116] Key type lgssc registered [ 3188.918230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3191.311471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3191.317269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.472897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.713049] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3196.412232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.535502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3197.714617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3199.312267] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3200.457897] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3202.630958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3202.745877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.554846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3206.207516] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.062890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3226.591886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3226.596937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.605695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.619527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.620181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.620518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.622750] Lustre: Skipped 1 previous similar message [ 3231.181179] Lustre: server umount lustre-OST0000 complete [ 3232.736597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.736886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3232.737264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.740469] Lustre: Skipped 1 previous similar message [ 3237.856620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.858883] Lustre: Skipped 2 previous similar messages [ 3238.486915] Lustre: server umount lustre-MDT0000 complete [ 3239.658143] LustreError: 195711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596185 with bad export cookie 6572329887387305205 [ 3239.661295] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.661729] LustreError: 195711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3239.790098] Lustre: server umount lustre-MDT0001 complete [ 3243.100293] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3244.077744] Key type lgssc unregistered [ 3244.204363] LNet: 198336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.206801] LNetError: 198336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.215267] LNet: Removed LNI 192.168.204.131@tcp [ 3244.488337] Key type .llcrypt unregistered [ 3244.489481] Key type ._llcrypt unregistered [ 3251.693737] Key type ._llcrypt registered [ 3251.694779] Key type .llcrypt registered [ 3251.729189] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3252.140671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3252.156320] alg: No test for adler32 (adler32-zlib) [ 3253.005837] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3253.082885] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3254.663090] Key type lgssc registered [ 3255.015932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.434898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.441713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.596870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.815232] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3262.465185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.600276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.824841] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3265.502851] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3266.617542] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3268.790472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3268.915770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.685928] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3272.375412] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.238373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3292.639860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.642773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.646407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.667381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.667695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.674848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.165489] Lustre: server umount lustre-OST0000 complete [ 3298.784479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.784573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3298.788640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.792376] Lustre: Skipped 2 previous similar messages [ 3303.904213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3303.906516] Lustre: Skipped 1 previous similar message [ 3304.473040] Lustre: server umount lustre-MDT0000 complete [ 3305.676231] LustreError: 201885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596251 with bad export cookie 16484524450679074852 [ 3305.679737] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3305.681097] LustreError: 201885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3305.799977] Lustre: server umount lustre-MDT0001 complete [ 3309.260194] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3310.238213] Key type lgssc unregistered [ 3310.374359] LNet: 202907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.376720] LNetError: 202907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.387316] LNet: Removed LNI 192.168.204.131@tcp [ 3310.666218] Key type .llcrypt unregistered [ 3310.667350] Key type ._llcrypt unregistered [ 3317.575523] Key type ._llcrypt registered [ 3317.577131] Key type .llcrypt registered [ 3317.613439] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3317.928308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3317.989891] alg: No test for adler32 (adler32-zlib) [ 3318.846586] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3318.933939] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3320.511172] Key type lgssc registered [ 3320.860106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3323.406896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3323.414873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.566542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.808642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3328.624298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.750468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.053566] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3331.827283] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3333.033818] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3335.217988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3335.337411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.128576] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3338.834374] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3348.648992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3351.796785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3351.823372] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3351.921438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.680897] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3355.472473] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3356.062759] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3366.675652] Lustre: Permanently deactivating lustre-OST0001 [ 3366.682115] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3376.117659] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3379.586173] Lustre: Permanently reactivating lustre-OST0001 [ 3379.591534] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3379.593478] Lustre: Skipped 2 previous similar messages [ 3384.825585] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.829820] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3384.832257] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3384.836368] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3384.848107] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3387.876203] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.876826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3387.881856] Lustre: Skipped 2 previous similar messages [ 3392.764386] Lustre: server umount lustre-OST0001 complete [ 3392.995741] LustreError: 205727:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.000375] LustreError: 205727:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3395.039766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.042368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.046169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3395.047716] Lustre: Skipped 1 previous similar message [ 3396.064107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.067367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.112904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3398.114533] LustreError: 205726:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.116061] Lustre: Skipped 1 previous similar message [ 3398.121339] LustreError: 205726:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3403.233407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.234415] LustreError: 207256:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3403.235798] Lustre: Skipped 1 previous similar message [ 3403.239855] LustreError: 207256:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3408.351108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3408.354592] LustreError: 205725:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.359940] LustreError: 205725:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 3408.409749] Lustre: server umount lustre-OST0000 complete [ 3410.911954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.915481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.921020] Lustre: Skipped 1 previous similar message [ 3413.472205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.474177] Lustre: Skipped 4 previous similar messages [ 3415.768073] Lustre: server umount lustre-MDT0000 complete [ 3416.963276] LustreError: 204113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596362 with bad export cookie 9435112874516566293 [ 3416.966685] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3416.967595] LustreError: 204113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3417.084043] Lustre: server umount lustre-MDT0001 complete [ 3420.465013] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3421.424368] Key type lgssc unregistered [ 3421.563310] LNet: 208769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3421.565595] LNetError: 208769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.573305] LNet: Removed LNI 192.168.204.131@tcp [ 3421.843424] Key type .llcrypt unregistered [ 3421.845295] Key type ._llcrypt unregistered [ 3434.831917] Key type ._llcrypt registered [ 3434.832925] Key type .llcrypt registered [ 3434.866247] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3435.209460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3435.245949] alg: No test for adler32 (adler32-zlib) [ 3436.091903] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3436.171074] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3437.751122] Key type lgssc registered [ 3438.101806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3440.674942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3440.681710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.772733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3441.781194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3441.843460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3443.136094] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3446.301448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3446.334412] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3446.344732] Lustre: 210931:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3446.438029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.856615] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3449.648568] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3450.815560] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3453.018536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3453.086232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3453.146005] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.217861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3454.957904] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3456.589428] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3458.098125] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3462.580431] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3471.196532] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3482.138763] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3490.742407] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3507.916301] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3507.918634] Lustre: Skipped 1 previous similar message [ 3547.852961] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3547.855825] Lustre: Skipped 4 previous similar messages [ 3614.114928] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3614.117609] Lustre: Skipped 8 previous similar messages [ 3628.513500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3628.514716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.516827] LustreError: Skipped 1 previous similar message [ 3628.520876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3628.523833] Lustre: Skipped 1 previous similar message [ 3632.711502] Lustre: server umount lustre-OST0000 complete [ 3636.191903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3636.194837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.199649] Lustre: Skipped 1 previous similar message [ 3636.201270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.752722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3638.753485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.024842] Lustre: server umount lustre-MDT0000 complete [ 3641.198247] LustreError: 213406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596586 with bad export cookie 12025587461486055681 [ 3641.200126] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3641.202515] LustreError: 213406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3641.317445] Lustre: server umount lustre-MDT0001 complete [ 3644.676051] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3645.711187] Key type lgssc unregistered [ 3645.840319] LNet: 215416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3645.843657] LNetError: 215416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3645.852425] LNet: Removed LNI 192.168.204.131@tcp [ 3646.119811] Key type .llcrypt unregistered [ 3646.120977] Key type ._llcrypt unregistered [ 3654.357345] Key type ._llcrypt registered [ 3654.358452] Key type .llcrypt registered [ 3654.391676] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3654.719044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3654.761194] alg: No test for adler32 (adler32-zlib) [ 3655.603630] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3655.683924] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3657.263138] Key type lgssc registered [ 3657.595962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3659.991976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3659.998060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3661.161400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3662.395911] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3665.042571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3665.165653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3666.457458] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3668.207584] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3669.375834] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3671.559043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.680658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.502045] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3675.265482] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3686.054958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3701.729410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.729870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.731229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.738772] Lustre: Skipped 1 previous similar message [ 3706.850492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.855260] Lustre: Skipped 2 previous similar messages [ 3707.180266] Lustre: server umount lustre-OST0000 complete [ 3711.455878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3711.458039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.462475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.088041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3717.090896] Lustre: Skipped 2 previous similar messages [ 3722.208882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.213059] Lustre: Skipped 1 previous similar message [ 3723.231090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3723.288296] Lustre: server umount lustre-MDT0000 complete [ 3724.618978] LustreError: 216767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1777596669 with bad export cookie 11970472590592885429 [ 3724.622830] LustreError: 216767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3724.622846] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.745720] Lustre: server umount lustre-MDT0001 complete [ 3728.984240] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3729.977538] Key type lgssc unregistered [ 3730.104427] LNet: 220142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3730.106800] LNetError: 220142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3730.114261] LNet: Removed LNI 192.168.204.131@tcp [ 3730.379243] Key type .llcrypt unregistered [ 3730.380935] Key type ._llcrypt unregistered [ 3738.476869] Key type ._llcrypt registered [ 3738.477816] Key type .llcrypt registered [ 3738.512897] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3738.837628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3738.873907] alg: No test for adler32 (adler32-zlib) [ 3739.723396] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 3739.805386] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3741.383188] Key type lgssc registered [ 3741.720515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4008.195130] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4009.229626] Key type lgssc unregistered [ 4009.368425] LNet: 222111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4009.370840] LNetError: 222111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4009.382438] LNet: Removed LNI 192.168.204.131@tcp [ 4009.718447] Key type .llcrypt unregistered [ 4009.719566] Key type ._llcrypt unregistered [ 4018.049761] Key type ._llcrypt registered [ 4018.050769] Key type .llcrypt registered [ 4018.089444] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4024.292669] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4024.690162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.701222] alg: No test for adler32 (adler32-zlib) [ 4025.551371] Lustre: Lustre: Build Version: 2.17.52_126_gb75093b [ 4025.641758] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4027.223160] Key type lgssc registered [ 4027.574668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.482979] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 3899 sec ========== 20:56:21 (1777596981) [ 4038.005032] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:56:22 (1777596982) === [ 4039.062723] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:56:23 (1777596983) === [ 4051.836043] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4052.826677] Key type lgssc unregistered [ 4052.948309] LNet: 227263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4052.950529] LNetError: 227263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4052.962321] LNet: Removed LNI 192.168.204.131@tcp [ 4053.257507] Key type .llcrypt unregistered [ 4053.258899] Key type ._llcrypt unregistered