[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 568019622 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002070] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011120] LSM: Security Framework initializing [ 0.012053] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.023604] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028269] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031058] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032120] *** VALIDATE tmpfs *** [ 0.034352] *** VALIDATE proc *** [ 0.036121] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.039250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043033] Spectre V2 : User space: Vulnerable [ 0.044015] Speculative Store Bypass: Vulnerable [ 0.047406] debug: unmapping init [mem 0xffffffff9ca59000-0xffffffff9ca60fff] [ 0.049261] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050902] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051026] ... version: 2 [ 0.052015] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054014] ... value mask: 0000ffffffffffff [ 0.055016] ... max period: 00007fffffffffff [ 0.056016] ... fixed-purpose events: 3 [ 0.057013] ... event mask: 000000070000000f [ 0.058373] rcu: Hierarchical SRCU implementation. [ 0.060733] smp: Bringing up secondary CPUs ... [ 0.061677] x86: Booting SMP configuration: [ 0.062028] .... node #0, CPUs: #1 #2 #3 [ 0.073085] smp: Brought up 1 node, 4 CPUs [ 0.075030] smpboot: Max logical packages: 1 [ 0.076017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124019] node 0 deferred pages initialised in 46ms [ 0.128239] devtmpfs: initialized [ 0.129320] x86/mm: Memory block size: 128MB [ 0.136407] gcov: version magic: 0x41383552 [ 0.139433] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144122] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149407] pinctrl core: initialized pinctrl subsystem [ 0.151167] [ 0.152010] ************************************************************* [ 0.155012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157012] ** ** [ 0.159013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161016] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.164015] ** IOMMU data structures, which may compromise security on ** [ 0.166014] ** your system. ** [ 0.169014] ** ** [ 0.170010] ** If you see this message and you are not debugging the ** [ 0.173014] ** kernel, report this immediately to your vendor! ** [ 0.175010] ** ** [ 0.179288] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182011] ************************************************************* [ 0.186170] NET: Registered protocol family 16 [ 0.189511] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194420] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201156] cpuidle: using governor menu [ 0.203028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206591] PCI: Using configuration type 1 for base access [ 0.209162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224094] cryptd: max_cpu_qlen set to 1000 [ 0.226701] ACPI: Added _OSI(Module Device) [ 0.227000] ACPI: Added _OSI(Processor Device) [ 0.229012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236581] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.245829] ACPI: Interpreter enabled [ 0.246077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247016] ACPI: Using IOAPIC for interrupt routing [ 0.248113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249975] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261861] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276732] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284903] acpiphp: Slot [2] registered [ 0.287196] acpiphp: Slot [5] registered [ 0.289301] acpiphp: Slot [6] registered [ 0.292088] acpiphp: Slot [7] registered [ 0.294280] acpiphp: Slot [8] registered [ 0.297119] acpiphp: Slot [9] registered [ 0.300322] acpiphp: Slot [10] registered [ 0.302133] acpiphp: Slot [3] registered [ 0.304142] acpiphp: Slot [4] registered [ 0.305000] acpiphp: Slot [11] registered [ 0.307158] acpiphp: Slot [12] registered [ 0.311120] acpiphp: Slot [13] registered [ 0.313136] acpiphp: Slot [14] registered [ 0.317149] acpiphp: Slot [15] registered [ 0.321137] acpiphp: Slot [16] registered [ 0.324156] acpiphp: Slot [17] registered [ 0.329134] acpiphp: Slot [18] registered [ 0.331147] acpiphp: Slot [19] registered [ 0.334258] acpiphp: Slot [20] registered [ 0.336120] acpiphp: Slot [21] registered [ 0.338128] acpiphp: Slot [22] registered [ 0.340113] acpiphp: Slot [23] registered [ 0.342136] acpiphp: Slot [24] registered [ 0.345131] acpiphp: Slot [25] registered [ 0.346142] acpiphp: Slot [26] registered [ 0.349162] acpiphp: Slot [27] registered [ 0.351530] acpiphp: Slot [28] registered [ 0.354188] acpiphp: Slot [29] registered [ 0.356125] acpiphp: Slot [30] registered [ 0.358149] acpiphp: Slot [31] registered [ 0.359184] PCI host bridge to bus 0000:00 [ 0.361021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.363026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.369027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.372030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.375030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.377175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.381348] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385584] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.397015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.404824] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.410030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.417778] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.421052] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.423147] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.429015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.457018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.467017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474864] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.485017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.499027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535042] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.548011] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558040] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.572675] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.598018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.619973] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.632021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.639020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.658024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.674000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.688022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.703027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.715000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.728384] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.740021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.749019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.768025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779551] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.787017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.815031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.824999] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.825163] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.827448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.828390] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.835112] iommu: Default domain type: Passthrough [ 0.836461] SCSI subsystem initialized [ 0.838129] ACPI: bus type USB registered [ 0.839124] usbcore: registered new interface driver usbfs [ 0.840107] usbcore: registered new interface driver hub [ 0.842068] usbcore: registered new device driver usb [ 0.843150] pps_core: LinuxPPS API ver. 1 registered [ 0.845011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.847063] PTP clock support registered [ 0.849138] EDAC MC: Ver: 3.0.0 [ 0.851177] PCI: Using ACPI for IRQ routing [ 0.852603] NetLabel: Initializing [ 0.853010] NetLabel: domain hash size = 128 [ 0.854042] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.856088] NetLabel: unlabeled traffic allowed by default [ 0.858276] vgaarb: loaded [ 0.859280] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.860011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.866000] clocksource: Switched to clocksource kvm-clock [ 1.076194] VFS: Disk quotas dquot_6.6.0 [ 1.088783] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.105201] *** VALIDATE ramfs *** [ 1.108174] *** VALIDATE hugetlbfs *** [ 1.111170] pnp: PnP ACPI init [ 1.127212] pnp: PnP ACPI: found 6 devices [ 1.154867] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.165936] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.173637] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.175785] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.182764] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.188936] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.195958] NET: Registered protocol family 2 [ 1.208457] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.226214] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.232540] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.248066] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.255748] TCP: Hash tables configured (established 65536 bind 65536) [ 1.282795] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.292626] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.299883] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.310570] NET: Registered protocol family 1 [ 1.338803] RPC: Registered named UNIX socket transport module. [ 1.346333] RPC: Registered udp transport module. [ 1.349405] RPC: Registered tcp transport module. [ 1.352541] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.357928] NET: Registered protocol family 44 [ 1.361816] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.367858] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.376400] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.380200] PCI: CLS 0 bytes, default 64 [ 1.382562] Unpacking initramfs... [ 4.758000] debug: unmapping init [mem 0xffff99403cc54000-0xffff99403ffbffff] [ 4.763718] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.766437] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.770786] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.499890] Initialise system trusted keyrings [ 5.501805] Key type blacklist registered [ 5.506492] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.516749] zbud: loaded [ 5.519981] *** VALIDATE nfs *** [ 5.521275] *** VALIDATE nfs4 *** [ 5.523036] pstore: using deflate compression [ 5.527284] Platform Keyring initialized [ 5.643735] NET: Registered protocol family 38 [ 5.646029] Key type asymmetric registered [ 5.647865] Asymmetric key parser 'x509' registered [ 5.650607] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.654492] io scheduler mq-deadline registered [ 5.656485] io scheduler kyber registered [ 5.658776] io scheduler bfq registered [ 5.660928] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.665169] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.668690] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.672387] ACPI: Power Button [PWRF] [ 5.678613] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.687438] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.710829] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.721591] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.739768] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.770524] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.800409] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.808942] Non-volatile memory driver v1.3 [ 5.810871] Linux agpgart interface v0.103 [ 5.842798] virtio_blk virtio1: [vda] 136576 512-byte logical blocks (69.9 MB/66.7 MiB) [ 5.846296] vda: detected capacity change from 0 to 69926912 [ 5.869507] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.873451] vdb: detected capacity change from 0 to 1073741824 [ 5.915643] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.924105] vdc: detected capacity change from 0 to 2621440000 [ 5.950691] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.954707] vdd: detected capacity change from 0 to 2621440000 [ 5.982823] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.986512] vde: detected capacity change from 0 to 4294967296 [ 6.019524] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.023335] vdf: detected capacity change from 0 to 4294967296 [ 6.045640] libphy: Fixed MDIO Bus: probed [ 6.061862] usbcore: registered new interface driver usbserial_generic [ 6.069051] usbserial: USB Serial support registered for generic [ 6.071051] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.081529] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.087496] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.094452] mousedev: PS/2 mouse device common for all mice [ 6.099956] rtc_cmos 00:05: RTC can wake from S4 [ 6.104295] rtc_cmos 00:05: registered as rtc0 [ 6.106684] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.111390] intel_pstate: CPU model not supported [ 6.114773] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.119572] hid: raw HID events driver (C) Jiri Kosina [ 6.127118] usbcore: registered new interface driver usbhid [ 6.129681] usbhid: USB HID core driver [ 6.132501] drop_monitor: Initializing network drop monitor service [ 6.136605] Initializing XFRM netlink socket [ 6.140976] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.147680] NET: Registered protocol family 10 [ 6.150121] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.154570] Segment Routing with IPv6 [ 6.154614] NET: Registered protocol family 17 [ 6.159237] mpls_gso: MPLS GSO support [ 6.168682] RAS: Correctable Errors collector initialized. [ 6.171363] AVX version of gcm_enc/dec engaged. [ 6.173488] AES CTR mode by8 optimization enabled [ 6.304707] sched_clock: Marking stable (6304617833, 0)->(7355336779, -1050718946) [ 6.310674] registered taskstats version 1 [ 6.313785] Loading compiled-in X.509 certificates [ 6.320106] zswap: loaded using pool lzo/zbud [ 6.355873] Key type big_key registered [ 6.376867] Key type encrypted registered [ 6.380767] ima: No TPM chip found, activating TPM-bypass! [ 6.388578] ima: Allocated hash algorithm: sha1 [ 6.390675] ima: No architecture policies found [ 6.393879] evm: Initialising EVM extended attributes: [ 6.398835] evm: security.selinux [ 6.401457] evm: security.ima [ 6.402794] evm: security.capability [ 6.404977] evm: HMAC attrs: 0x1 [ 6.409639] rtc_cmos 00:05: setting system clock to 2026-06-12 04:09:46 UTC (1781237386) [ 6.417628] debug: unmapping init [mem 0xffffffff9da03000-0xffffffff9dbfffff] [ 6.421686] debug: unmapping init [mem 0xffffffff9c782000-0xffffffff9ca58fff] [ 6.427069] Write protecting the kernel read-only data: 28672k [ 6.431277] debug: unmapping init [mem 0xffffffff9ae03000-0xffffffff9affffff] [ 6.434834] debug: unmapping init [mem 0xffffffff9b714000-0xffffffff9b7fffff] [ 6.506095] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.531625] systemd[1]: Detected virtualization kvm. [ 6.538765] systemd[1]: Detected architecture x86-64. [ 6.541927] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.579024] systemd[1]: No hostname configured. [ 6.581043] systemd[1]: Set hostname to . [ 6.583924] random: systemd: uninitialized urandom read (16 bytes read) [ 6.585714] systemd[1]: Initializing machine ID from random generator. [ 6.678433] random: ln: uninitialized urandom read (6 bytes read) [ 6.862568] random: systemd: uninitialized urandom read (16 bytes read) [ 6.866252] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.871630] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.901001] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.662974] device-mapper: uevent: version 1.0.3 [ 8.674115] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.063271] virtio_net virtio0 ens2: renamed from eth0 [ 11.113307] random: fast init done [ 11.791108] scsi host0: ata_piix [ 11.917835] scsi host1: ata_piix [ 11.919632] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.923558] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.923620] random: crng init done [ 16.928880] random: 7 urandom warning(s) missed due to ratelimiting [ 18.280320] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 20.181853] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.055749] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.893135] SELinux: Disabled at runtime. [ 24.054490] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.071554] systemd[1]: Detected virtualization kvm. [ 24.075468] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.339221] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.348236] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.360692] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.370720] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.382487] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.391031] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.399578] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 25.571076] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 26.573310] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.490365] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.498666] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.987115] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.123643] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 33.516802] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.563876] NFS: Registering the id_resolver key type [ 34.567868] Key type id_resolver registered [ 34.571865] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg259-server login: [ 57.353197] hrtimer: interrupt took 8188578 ns [ 90.798747] libcfs: loading out-of-tree module taints kernel. [ 90.862630] Key type ._llcrypt registered [ 90.868544] Key type .llcrypt registered [ 90.969808] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 109.671821] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 111.058090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.069253] alg: No test for adler32 (adler32-zlib) [ 112.358341] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 113.118673] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 114.863174] Key type lgssc registered [ 116.839988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.207808] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 178.085509] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 191.494513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.529204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.983550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.059664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.286840] Lustre: lustre-MDT0000: new disk, initializing [ 193.468542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.499917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.194931] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.207860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.399877] Lustre: 6510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 214.446803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 214.452941] Lustre: Skipped 1 previous similar message [ 214.551628] Lustre: lustre-MDT0001: new disk, initializing [ 214.641596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 214.667792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 214.680098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.141646] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.937677] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.196661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.789396] Lustre: lustre-OST0000: new disk, initializing [ 239.794765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.812713] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.931121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.448769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.461463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.578561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.054224] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.054475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.212790] Lustre: lustre-OST0001: new disk, initializing [ 263.216744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.222545] Lustre: 9469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.315754] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 268.881122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 268.894868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 268.946347] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.233393] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.404299] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.072644] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 295.082577] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.362487] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing yml_node [ 306.713496] Lustre: DEBUG MARKER: Client: 2.17.53.78 [ 309.408644] Lustre: DEBUG MARKER: MDS: 2.17.53.78 [ 312.306721] Lustre: DEBUG MARKER: OSS: 2.17.53.78 [ 314.258450] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 00:14:51 EDT 2026 [ 333.633034] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 335.279544] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.739922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 352.751079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.784576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.330152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.338593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.346351] Lustre: Skipped 2 previous similar messages [ 356.355068] Lustre: Skipped 2 previous similar messages [ 357.598383] Lustre: server umount lustre-MDT0000 complete [ 362.104509] LustreError: 9468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237742 with bad export cookie 9647798511550262832 [ 362.110582] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.116795] LustreError: 9468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 366.561073] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.577410] LustreError: 10981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.578224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.578235] Lustre: Skipped 1 previous similar message [ 366.579150] Lustre: Skipped 1 previous similar message [ 366.603326] LustreError: 10981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 367.586145] LustreError: 10838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.618306] LustreError: 10838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 368.565744] Lustre: server umount lustre-MDT0001 complete [ 372.549582] Lustre: server umount lustre-OST0000 complete [ 376.700083] Lustre: server umount lustre-OST0001 complete [ 384.815305] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 395.126585] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 431.456073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.755790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 431.799737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 431.934380] Lustre: lustre-MDT0000: new disk, initializing [ 432.038140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.079479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.919965] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 447.114261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.249389] Lustre: 14986:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 447.328545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 447.338240] Lustre: Skipped 1 previous similar message [ 447.495390] Lustre: lustre-MDT0001: new disk, initializing [ 447.675853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 447.778692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 447.805712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.867775] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 459.180685] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 462.797251] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 470.655198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.033265] Lustre: lustre-OST0000: new disk, initializing [ 471.036920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.044424] Lustre: 16727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.204630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.071513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.078911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.172374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 478.283782] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 487.354570] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 492.908278] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 493.139827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 497.800884] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.064503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 503.780980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 503.786706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.806094] LustreError: Skipped 1 previous similar message [ 503.808722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 503.829369] Lustre: Skipped 1 previous similar message [ 503.856810] Lustre: Skipped 4 previous similar messages [ 506.314893] Lustre: server umount lustre-OST0000 complete [ 514.020420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.022482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.040165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.053540] Lustre: Skipped 1 previous similar message [ 514.083355] Lustre: Skipped 1 previous similar message [ 517.124174] Lustre: server umount lustre-MDT0000 complete [ 519.136097] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.158626] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 520.471213] LustreError: 15763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237900 with bad export cookie 9647798511550264365 [ 520.475064] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 520.488841] LustreError: 15763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 520.860664] Lustre: server umount lustre-MDT0001 complete [ 524.257798] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 00:18:21 (1781237901) [ 531.038151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.672697] LustreError: 18842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.752949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.180981] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 537.063172] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.997250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.278677] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 551.385362] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 556.968428] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 561.133788] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.473192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.910588] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 583.860714] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 596.969844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.973987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.982871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 596.992911] Lustre: Skipped 1 previous similar message [ 597.037195] LustreError: Skipped 1 previous similar message [ 598.746674] Lustre: server umount lustre-OST0000 complete [ 609.624996] Lustre: server umount lustre-MDT0000 complete [ 612.329289] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.355401] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 613.839102] LustreError: 18823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237993 with bad export cookie 9647798511550265247 [ 613.850234] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.865323] LustreError: 18823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 614.480187] Lustre: server umount lustre-MDT0001 complete [ 628.493299] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 632.879970] Key type lgssc unregistered [ 633.191199] LNet: 22018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.203432] LNetError: 22018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.223553] LNet: Removed LNI 192.168.202.159@tcp [ 634.059346] Key type .llcrypt unregistered [ 634.063941] Key type ._llcrypt unregistered [ 662.805590] Key type ._llcrypt registered [ 662.807542] Key type .llcrypt registered [ 662.929365] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 664.706537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 664.782447] alg: No test for adler32 (adler32-zlib) [ 665.847109] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 666.022710] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 667.799270] Key type lgssc registered [ 669.071218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 676.194952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 676.234160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 677.780413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 682.597880] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 687.080065] LustreError: 23422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.100713] LustreError: 23422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 690.159454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.393541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 694.616756] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 699.586679] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 702.528796] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 708.804492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.246885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.705202] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 720.786161] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 722.557719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 736.225796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 736.238852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.250061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 738.027831] Lustre: server umount lustre-OST0000 complete [ 741.855979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 741.865602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.887422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.393038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 743.395064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.512965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 748.518384] Lustre: Skipped 2 previous similar messages [ 753.633992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.193639] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 756.319065] LustreError: 26305:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (000000005d2052b0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 756.372747] Lustre: server umount lustre-MDT0000 complete [ 758.755045] LustreError: 23423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.774114] LustreError: 23423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 760.350171] LustreError: 23401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238140 with bad export cookie 9860717610055289849 [ 760.354931] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.365664] LustreError: 23401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 760.808438] Lustre: server umount lustre-MDT0001 complete [ 771.681143] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 774.608161] Key type lgssc unregistered [ 774.917646] LNet: 27041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.925390] LNetError: 27041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.970629] LNet: Removed LNI 192.168.202.159@tcp [ 775.943694] Key type .llcrypt unregistered [ 775.950621] Key type ._llcrypt unregistered [ 797.783699] Key type ._llcrypt registered [ 797.788067] Key type .llcrypt registered [ 797.934522] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 799.109654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.203115] alg: No test for adler32 (adler32-zlib) [ 800.295435] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 800.655447] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 802.328854] Key type lgssc registered [ 803.341175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.151907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.186289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.846767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 817.818508] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 822.254392] LustreError: 28417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.287905] LustreError: 28417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 826.381947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.812245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 831.216683] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 836.494607] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 839.536821] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 853.803741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 854.249222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 857.339641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 859.934883] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 865.271681] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 877.033693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.039940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 877.063451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.149091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 882.151974] Lustre: Skipped 1 previous similar message [ 887.265878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.276871] Lustre: Skipped 1 previous similar message [ 891.362796] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 891.587819] Lustre: server umount lustre-OST0000 complete [ 897.505449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.509994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.539065] Lustre: Skipped 2 previous similar messages [ 897.549740] Lustre: Skipped 1 previous similar message [ 902.630961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.646543] Lustre: Skipped 1 previous similar message [ 909.279399] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 909.380874] Lustre: server umount lustre-MDT0000 complete [ 912.444102] LustreError: 28396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238292 with bad export cookie 14741927371539389090 [ 912.448360] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 912.453860] LustreError: 28396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 912.726296] Lustre: server umount lustre-MDT0001 complete [ 923.426481] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 926.960810] Key type lgssc unregistered [ 927.370617] LNet: 31940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.381332] LNetError: 31940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.407290] LNet: Removed LNI 192.168.202.159@tcp [ 928.438749] Key type .llcrypt unregistered [ 928.443830] Key type ._llcrypt unregistered [ 953.988129] Key type ._llcrypt registered [ 953.990228] Key type .llcrypt registered [ 954.055848] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 955.291787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 955.316680] alg: No test for adler32 (adler32-zlib) [ 956.528684] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 956.910376] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 958.695654] Key type lgssc registered [ 959.810668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 970.643994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 970.693747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.457209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.762753] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 981.988451] LustreError: 33337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.010881] LustreError: 33337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 985.692398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.304157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.078712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 997.246179] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1000.919883] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1007.466345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.787168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.752224] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1017.024746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1019.910024] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1030.116612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1030.126600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.143515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.682221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.688460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.688747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1032.706392] Lustre: Skipped 1 previous similar message [ 1035.075519] Lustre: server umount lustre-OST0000 complete [ 1042.401847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1042.417302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.437958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1045.925707] Lustre: server umount lustre-MDT0000 complete [ 1048.033619] LustreError: 34048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.050638] LustreError: 34048:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1050.299604] LustreError: 33316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238430 with bad export cookie 3477455735269193679 [ 1050.311587] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1050.634810] Lustre: server umount lustre-MDT0001 complete [ 1062.758554] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1065.368329] Key type lgssc unregistered [ 1065.706574] LNet: 36515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.714504] LNetError: 36515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.741402] LNet: Removed LNI 192.168.202.159@tcp [ 1066.570752] Key type .llcrypt unregistered [ 1066.576038] Key type ._llcrypt unregistered [ 1092.993292] Key type ._llcrypt registered [ 1092.995695] Key type .llcrypt registered [ 1093.100702] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1094.567643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1094.643324] alg: No test for adler32 (adler32-zlib) [ 1095.852339] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1096.163272] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1097.927271] Key type lgssc registered [ 1099.260089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.131552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.160911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.738726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.055239] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1120.293838] LustreError: 37910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.324706] LustreError: 37910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1125.354113] LustreError: 37911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.827432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.291025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1133.361369] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1139.416945] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1143.660475] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1151.037267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.487798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.562958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1157.802580] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1163.277570] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1169.888493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1169.898210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.920959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.914222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1170.916050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1170.917355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1175.960170] Lustre: server umount lustre-OST0000 complete [ 1196.001052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1196.011635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.040865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.053467] Lustre: Skipped 2 previous similar messages [ 1200.597485] Lustre: server umount lustre-MDT0000 complete [ 1202.658753] LustreError: 37907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.686805] LustreError: 37907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1204.072334] LustreError: 37891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238584 with bad export cookie 10774260422263139601 [ 1204.095267] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.360823] Lustre: server umount lustre-MDT0001 complete [ 1221.680327] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1224.183600] Key type lgssc unregistered [ 1224.444190] LNet: 41259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.458377] LNetError: 41259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.510955] LNet: Removed LNI 192.168.202.159@tcp [ 1226.379428] Key type .llcrypt unregistered [ 1226.381485] Key type ._llcrypt unregistered [ 1246.846659] Key type ._llcrypt registered [ 1246.850702] Key type .llcrypt registered [ 1246.981712] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1247.766245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.862701] alg: No test for adler32 (adler32-zlib) [ 1249.021279] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1249.290599] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1250.928795] Key type lgssc registered [ 1252.313223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.661298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.678232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.128980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.357862] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1272.312037] LustreError: 42635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.344555] LustreError: 42635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1276.798850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.136501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.925258] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1287.766739] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.955989] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1300.484641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1301.050517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.996418] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1307.185151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1313.945092] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1322.478904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.483636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.509743] Lustre: Skipped 1 previous similar message [ 1322.535534] Lustre: Skipped 2 previous similar messages [ 1327.013852] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1332.103057] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.59@tcp (stopping) [ 1332.111459] Lustre: Skipped 3 previous similar messages [ 1336.287139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1336.600493] Lustre: server umount lustre-MDT0000 complete [ 1337.217144] LustreError: 42632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.805498] LustreError: 42616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238720 with bad export cookie 16914281972631106552 [ 1340.813969] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.832158] LustreError: 42616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1342.338886] LustreError: 44253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.350032] LustreError: 44253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1342.363773] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 1342.368362] Lustre: Skipped 3 previous similar messages [ 1342.957883] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.975703] Lustre: Skipped 1 previous similar message [ 1345.955771] LustreError: 42630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.976312] LustreError: 42630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1347.457552] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.59@tcp (stopping) [ 1347.469653] Lustre: Skipped 2 previous similar messages [ 1350.051359] LustreError: 42630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.065853] LustreError: 42630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1355.231179] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1355.555318] Lustre: server umount lustre-MDT0001 complete [ 1372.383872] Lustre: server umount lustre-OST0000 complete [ 1391.114418] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1394.812628] Key type lgssc unregistered [ 1395.137882] LNet: 46146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.151538] LNetError: 46146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.200768] LNet: Removed LNI 192.168.202.159@tcp [ 1397.399534] Key type .llcrypt unregistered [ 1397.401539] Key type ._llcrypt unregistered [ 1423.381362] Key type ._llcrypt registered [ 1423.386029] Key type .llcrypt registered [ 1423.514431] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1424.482776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1424.682768] alg: No test for adler32 (adler32-zlib) [ 1425.834627] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1426.167965] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1427.937895] Key type lgssc registered [ 1429.266467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.412676] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.427765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1453.025783] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1458.146841] LustreError: 47518:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 1484.831203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.459959] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1498.270828] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1666.718524] Lustre: server umount lustre-OST0000 complete [ 1684.516663] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1687.375619] Key type lgssc unregistered [ 1687.870235] LNet: 49062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.882180] LNetError: 49062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.903742] LNet: Removed LNI 192.168.202.159@tcp [ 1688.906971] Key type .llcrypt unregistered [ 1688.907965] Key type ._llcrypt unregistered [ 1719.708526] Key type ._llcrypt registered [ 1719.719095] Key type .llcrypt registered [ 1719.884663] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1721.978416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.018717] alg: No test for adler32 (adler32-zlib) [ 1723.359860] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1723.756745] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1725.471200] Key type lgssc registered [ 1727.486527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1738.375830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1738.411052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.944847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.632721] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1749.477637] LustreError: 50463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1749.518343] LustreError: 50463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1754.595082] LustreError: 50462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.839263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.229089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.787031] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1766.691554] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1770.363954] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1777.684264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1777.949534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.120529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1784.664228] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1791.070318] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1794.533413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.541397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1794.541866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1794.564113] Lustre: Skipped 1 previous similar message [ 1799.674268] Lustre: server umount lustre-OST0000 complete [ 1804.780667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.788340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1804.806043] Lustre: Skipped 1 previous similar message [ 1804.821226] Lustre: Skipped 2 previous similar messages [ 1809.891379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1809.906563] Lustre: Skipped 1 previous similar message [ 1810.185053] Lustre: server umount lustre-MDT0000 complete [ 1813.986446] LustreError: 51173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.029913] LustreError: 51173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1814.861337] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239194 with bad export cookie 15111654516978616520 [ 1814.868662] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.872304] LustreError: 50443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1815.375649] Lustre: server umount lustre-MDT0001 complete [ 1828.369491] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 1832.312849] Key type lgssc unregistered [ 1832.772505] LNet: 53637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.812874] LNetError: 53637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.898690] LNet: Removed LNI 192.168.202.159@tcp [ 1835.235321] Key type .llcrypt unregistered [ 1835.240652] Key type ._llcrypt unregistered [ 1865.520548] Key type ._llcrypt registered [ 1865.524666] Key type .llcrypt registered [ 1865.699208] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 1867.266337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.291146] alg: No test for adler32 (adler32-zlib) [ 1868.867383] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1869.128456] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 1870.783837] Key type lgssc registered [ 1872.416172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1882.165941] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1882.202531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1897.762613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1902.879320] LustreError: 55006:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 1929.631417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.857894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1941.027418] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1947.526848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.026817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.118346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1952.954226] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1953.274140] LustreError: 55032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.295433] LustreError: 55032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1958.369318] LustreError: 55031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.504103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.907996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.262976] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 1974.782564] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1979.690665] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1983.467145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.478744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.506465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.985161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.994496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.007785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.513466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.524851] Lustre: Skipped 1 previous similar message [ 1988.852380] Lustre: server umount lustre-OST0000 complete [ 2014.177643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.192271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.216778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.071561] Lustre: server umount lustre-MDT0000 complete [ 2020.323432] LustreError: 56996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.358183] LustreError: 56996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2023.692500] LustreError: 55014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239403 with bad export cookie 1151023768727605829 [ 2023.695928] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.710045] LustreError: 55014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2030.138630] Lustre: server umount lustre-MDT0001 complete [ 2042.518265] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2045.748775] Key type lgssc unregistered [ 2046.418952] LNet: 58395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.427052] LNetError: 58395:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.446800] LNet: Removed LNI 192.168.202.159@tcp [ 2047.455805] Key type .llcrypt unregistered [ 2047.462226] Key type ._llcrypt unregistered [ 2072.603767] Key type ._llcrypt registered [ 2072.607060] Key type .llcrypt registered [ 2072.806209] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2074.115977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.139676] alg: No test for adler32 (adler32-zlib) [ 2075.156569] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2075.421245] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2077.151826] Key type lgssc registered [ 2078.550657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2087.229314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2087.260362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.807448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.520842] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2098.145120] LustreError: 59792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.175174] LustreError: 59792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2102.480638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.001990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.324676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2114.303513] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2119.008430] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2126.302237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2126.624699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.665546] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2133.806145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2137.934993] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2154.466268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.469587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.480137] LustreError: Skipped 1 previous similar message [ 2154.480888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.523425] Lustre: Skipped 1 previous similar message [ 2156.606857] Lustre: server umount lustre-OST0000 complete [ 2164.705059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.715150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.728710] Lustre: Skipped 1 previous similar message [ 2164.753282] Lustre: Skipped 2 previous similar messages [ 2166.989627] Lustre: server umount lustre-MDT0000 complete [ 2169.825180] LustreError: 59789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.841770] LustreError: 59789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2170.772746] LustreError: 59773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239550 with bad export cookie 15706214866464965667 [ 2170.778410] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.790879] LustreError: 59773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2171.165470] Lustre: server umount lustre-MDT0001 complete [ 2181.773192] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2184.695634] Key type lgssc unregistered [ 2185.040796] LNet: 62992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.054718] LNetError: 62992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.076929] LNet: Removed LNI 192.168.202.159@tcp [ 2186.034346] Key type .llcrypt unregistered [ 2186.038505] Key type ._llcrypt unregistered [ 2212.595816] Key type ._llcrypt registered [ 2212.598417] Key type .llcrypt registered [ 2212.712800] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2214.530836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2214.561309] alg: No test for adler32 (adler32-zlib) [ 2215.591441] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2215.786759] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2217.483321] Key type lgssc registered [ 2219.109875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2231.870496] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 00:46:48 (1781239608) [ 2240.303532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2240.342793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.959479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.901425] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2252.257946] LustreError: 64684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.290447] LustreError: 64684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2255.204667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.772203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.863683] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2266.612955] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2271.425843] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.020194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.439248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.889436] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2287.676987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2292.794930] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2299.733201] Lustre: Failing over lustre-MDT0000 [ 2300.100026] Lustre: server umount lustre-MDT0000 complete [ 2300.841774] LustreError: 64680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.920297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.930538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.940073] LustreError: 64685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.945339] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.922450] LustreError: 64679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.948484] LustreError: 64679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2309.283531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.394948] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.532183] Lustre: *** cfs_fail_loc=135, val=0*** [ 2309.708343] LustreError: 67429:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2309.721158] LustreError: 67429:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -2 [ 2309.735148] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2309.735148] [ 2309.742290] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2309.768388] LustreError: 67417:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2309.777161] LustreError: 67417:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2309.787428] LustreError: 67417:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2309.792439] LustreError: 67417:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2309.821936] Lustre: server umount lustre-MDT0000 complete [ 2309.823474] LustreError: 67417:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2311.043227] LustreError: 64679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.076131] LustreError: 64679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2316.622939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.722586] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.962790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.012290] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2320.830696] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2321.291384] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2322.430702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2322.457354] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2322.517657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2331.616516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2331.640671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.676232] Lustre: Skipped 1 previous similar message [ 2331.706735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2332.651659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2332.662184] Lustre: Skipped 1 previous similar message [ 2337.764391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.879740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2342.886602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.901962] Lustre: Skipped 2 previous similar messages [ 2343.209220] Lustre: server umount lustre-OST0000 complete [ 2348.002754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.008307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2348.013912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2348.020304] Lustre: Skipped 2 previous similar messages [ 2348.052874] Lustre: Skipped 1 previous similar message [ 2354.470625] Lustre: server umount lustre-MDT0000 complete [ 2358.255203] LustreError: 64684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.301522] LustreError: 64684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2359.802319] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239739 with bad export cookie 9856340865990308388 [ 2359.806275] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.811804] LustreError: 64666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2360.290740] Lustre: server umount lustre-MDT0001 complete [ 2371.167590] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2374.627606] Key type lgssc unregistered [ 2375.124645] LNet: 69175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.135870] LNetError: 69175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.149639] LNet: Removed LNI 192.168.202.159@tcp [ 2376.143785] Key type .llcrypt unregistered [ 2376.147069] Key type ._llcrypt unregistered [ 2401.482575] Key type ._llcrypt registered [ 2401.486699] Key type .llcrypt registered [ 2401.659507] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2403.424289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.724208] alg: No test for adler32 (adler32-zlib) [ 2405.070292] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2405.511561] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2407.319758] Key type lgssc registered [ 2409.423533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2418.892726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2418.937370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.336872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.407664] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2429.924377] LustreError: 70571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.945863] LustreError: 70571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2432.988716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.300363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2437.927756] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2444.729761] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2449.512589] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2459.884743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.489993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.665445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2468.917379] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2476.293846] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2482.963821] Lustre: Failing over lustre-MDT0000 [ 2483.188183] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.195368] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.201615] Lustre: Skipped 1 previous similar message [ 2483.230661] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2483.368793] Lustre: server umount lustre-MDT0000 complete [ 2488.298442] LustreError: 71549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.337537] LustreError: 71549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2493.410681] LustreError: 70571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.493707] LustreError: 70571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2494.885942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.178093] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2495.548149] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2495.843738] LustreError: 73321:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2495.859645] LustreError: 73321:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -9 [ 2495.888201] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2495.888201] [ 2495.906099] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2495.951602] LustreError: 73309:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2495.981062] LustreError: 73309:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 2495.997628] LustreError: 73309:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2496.018742] LustreError: 73309:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2496.052674] Lustre: server umount lustre-MDT0000 complete [ 2496.054797] LustreError: 73309:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2498.534553] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.564715] LustreError: 70572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2504.602346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.784542] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2504.810899] LustreError: 70563:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2505.210838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.333401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2510.214744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2510.321887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2510.350879] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2510.420873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2511.688348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2523.529528] Lustre: server umount lustre-OST0000 complete [ 2530.784341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2530.786637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.799174] Lustre: Skipped 2 previous similar messages [ 2530.800403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.907050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.921325] Lustre: Skipped 1 previous similar message [ 2539.999195] Lustre: 69883:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239904/real 1781239904] req@0000000060358060 x1867765288843264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781239920 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2540.052891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.079198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2540.087515] Lustre: Skipped 1 previous similar message [ 2540.960372] Lustre: 69885:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239905/real 1781239905] req@00000000cb6e8215 x1867765288843648/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781239921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2542.048824] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2542.198933] Lustre: server umount lustre-MDT0000 complete [ 2545.124934] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.149291] LustreError: 70565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2546.208294] Lustre: 69883:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781239910/real 1781239910] req@000000002df23ce1 x1867765288845824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781239926 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2546.575812] LustreError: 70550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239926 with bad export cookie 8918871358658980014 [ 2546.576807] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2546.585720] LustreError: 70550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2547.135256] Lustre: server umount lustre-MDT0001 complete [ 2559.727911] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2562.822165] Key type lgssc unregistered [ 2563.246829] LNet: 75073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.257429] LNetError: 75073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.273777] LNet: Removed LNI 192.168.202.159@tcp [ 2564.240363] Key type .llcrypt unregistered [ 2564.242324] Key type ._llcrypt unregistered [ 2580.069430] Key type ._llcrypt registered [ 2580.071855] Key type .llcrypt registered [ 2580.230547] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2580.561307] Key type .llcrypt unregistered [ 2580.563658] Key type ._llcrypt unregistered [ 2605.933857] Key type ._llcrypt registered [ 2605.935857] Key type .llcrypt registered [ 2606.077507] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2607.631112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2607.710024] alg: No test for adler32 (adler32-zlib) [ 2608.931617] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2609.222718] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2610.895895] Key type lgssc registered [ 2612.381489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2621.936730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2621.953539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.581898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.344761] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2633.699240] LustreError: 76941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.729174] LustreError: 76941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2637.500667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.076339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2642.514831] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2648.393668] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2652.652890] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2659.737593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2660.334492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.036956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2668.527719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2672.845240] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2683.877730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.887470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.924350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2688.629097] Lustre: server umount lustre-OST0000 complete [ 2694.112558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2694.114532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.125181] LustreError: Skipped 1 previous similar message [ 2694.148865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.151370] Lustre: Skipped 2 previous similar messages [ 2694.157793] Lustre: Skipped 1 previous similar message [ 2699.233576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.236299] Lustre: Skipped 2 previous similar messages [ 2699.703062] Lustre: server umount lustre-MDT0000 complete [ 2704.364636] LustreError: 76941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.392721] LustreError: 76941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2704.411677] LustreError: 76921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240084 with bad export cookie 7089285166431996460 [ 2704.414167] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.426631] LustreError: 76921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2704.863477] Lustre: server umount lustre-MDT0001 complete [ 2716.896195] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2720.026763] Key type lgssc unregistered [ 2720.351551] LNet: 80116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.370729] LNetError: 80116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2720.382235] LNet: Removed LNI 192.168.202.159@tcp [ 2721.262306] Key type .llcrypt unregistered [ 2721.265541] Key type ._llcrypt unregistered [ 2746.172660] Key type ._llcrypt registered [ 2746.177498] Key type .llcrypt registered [ 2746.287173] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2747.382598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2747.639449] alg: No test for adler32 (adler32-zlib) [ 2748.669694] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2748.929314] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2750.599247] Key type lgssc registered [ 2752.323682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2760.808106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2760.823747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.345183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.291409] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2772.454155] LustreError: 81512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.490799] LustreError: 81512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2773.485231] LustreError: 81513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.100989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.360451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.798540] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2788.284956] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2792.900109] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2800.048345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.450228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.602978] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2808.626940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2813.564303] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.581178] Lustre: server umount lustre-OST0000 complete [ 2828.770827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2828.793561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.823373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.414856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.420831] Lustre: Skipped 2 previous similar messages [ 2835.491945] Lustre: server umount lustre-MDT0000 complete [ 2839.076909] Lustre: 80823:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240203/real 1781240203] req@00000000deebbeb1 x1867765649532672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781240219 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2839.124638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.139386] Lustre: Skipped 1 previous similar message [ 2839.146135] LustreError: 81513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.772211] LustreError: 81492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240219 with bad export cookie 10510908799487846894 [ 2839.776506] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2839.799130] LustreError: 81492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2840.331457] Lustre: server umount lustre-MDT0001 complete [ 2852.120781] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2855.026196] Key type lgssc unregistered [ 2855.355644] LNet: 84689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2855.391832] LNetError: 84689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2855.412852] LNet: Removed LNI 192.168.202.159@tcp [ 2856.167331] Key type .llcrypt unregistered [ 2856.169750] Key type ._llcrypt unregistered [ 2880.067820] Key type ._llcrypt registered [ 2880.069924] Key type .llcrypt registered [ 2880.179285] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 2881.169817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2881.334221] alg: No test for adler32 (adler32-zlib) [ 2882.527953] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2882.751333] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 2884.479182] Key type lgssc registered [ 2885.614536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2894.282727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2894.305166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.968187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2901.777886] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2905.082907] LustreError: 86089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.128217] LustreError: 86089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2910.176351] LustreError: 86090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.323489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.675669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2916.182054] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2921.133782] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2924.149185] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2930.029921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2930.420781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2936.396560] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 2941.695867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2941.720084] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2954.727186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2954.733907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.742900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.282648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.287292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.316344] Lustre: Skipped 1 previous similar message [ 2960.929697] Lustre: server umount lustre-OST0000 complete [ 2967.523904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.542438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.641939] Lustre: server umount lustre-MDT0000 complete [ 2972.648202] LustreError: 86089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.681276] LustreError: 86089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2975.477593] LustreError: 86071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240355 with bad export cookie 12329607482890285074 [ 2975.485331] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2975.492434] LustreError: 86071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2975.870938] Lustre: server umount lustre-MDT0001 complete [ 2985.192742] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 2987.982815] Key type lgssc unregistered [ 2988.222900] LNet: 89283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2988.229084] LNetError: 89283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2988.242594] LNet: Removed LNI 192.168.202.159@tcp [ 2989.120327] Key type .llcrypt unregistered [ 2989.122789] Key type ._llcrypt unregistered [ 3017.385289] Key type ._llcrypt registered [ 3017.391306] Key type .llcrypt registered [ 3017.500295] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3018.495066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.637452] alg: No test for adler32 (adler32-zlib) [ 3019.719530] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3019.921692] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3021.551195] Key type lgssc registered [ 3022.622622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3029.425187] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3029.447421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.023595] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3050.144397] LustreError: 90673:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 3076.831838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.685819] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3087.817535] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3092.847510] Lustre: server umount lustre-OST0000 complete [ 3100.117380] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 01:01:17 (1781240477) [ 3106.942895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.212943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.974679] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3112.420024] LustreError: 92214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.458872] LustreError: 92214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3117.552627] LustreError: 92215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.242732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.595519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3122.050670] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3127.222902] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3130.816937] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3137.319138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.581331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.646216] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3148.949853] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3152.961520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3168.233364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3168.243262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3168.249664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.275299] Lustre: Skipped 1 previous similar message [ 3168.290933] Lustre: Skipped 1 previous similar message [ 3170.977146] Lustre: server umount lustre-OST0000 complete [ 3178.464258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3178.469119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3178.477958] Lustre: Skipped 1 previous similar message [ 3178.490031] Lustre: Skipped 1 previous similar message [ 3181.369756] Lustre: server umount lustre-MDT0000 complete [ 3183.585556] LustreError: 92214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3183.602742] LustreError: 92214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3184.926225] LustreError: 92195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240565 with bad export cookie 13456460632777738695 [ 3184.930263] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.936181] LustreError: 92195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3185.200863] Lustre: server umount lustre-MDT0001 complete [ 3195.181510] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3198.132426] Key type lgssc unregistered [ 3198.384724] LNet: 95549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3198.391557] LNetError: 95549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3198.413970] LNet: Removed LNI 192.168.202.159@tcp [ 3199.181527] Key type .llcrypt unregistered [ 3199.183087] Key type ._llcrypt unregistered [ 3234.983977] Key type ._llcrypt registered [ 3234.987636] Key type .llcrypt registered [ 3235.074649] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3236.498463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3236.611470] alg: No test for adler32 (adler32-zlib) [ 3237.770449] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3238.155677] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3239.863181] Key type lgssc registered [ 3241.295359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3247.922935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3247.951235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.506204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.188881] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3259.875690] LustreError: 97717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.890786] LustreError: 97717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3259.928184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.379339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.680301] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3269.019724] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3273.679076] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3280.836848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3281.207599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3286.634378] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3293.671673] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3294.522957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3304.935944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3304.941581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.951125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3304.962313] Lustre: Skipped 1 previous similar message [ 3304.988539] Lustre: Skipped 1 previous similar message [ 3309.771544] Lustre: server umount lustre-OST0000 complete [ 3315.178531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3315.197974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.208883] Lustre: Skipped 1 previous similar message [ 3315.231824] Lustre: Skipped 1 previous similar message [ 3320.290160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3320.301085] Lustre: Skipped 1 previous similar message [ 3320.898799] Lustre: server umount lustre-MDT0000 complete [ 3324.593562] LustreError: 97697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240704 with bad export cookie 9369681634286493993 [ 3324.598746] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3324.601980] LustreError: 97697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3324.980835] Lustre: server umount lustre-MDT0001 complete [ 3335.399809] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3338.126709] Key type lgssc unregistered [ 3338.420458] LNet: 100893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.432363] LNetError: 100893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3339.510646] LNet: Removed LNI 192.168.202.159@tcp [ 3340.503282] Key type .llcrypt unregistered [ 3340.505959] Key type ._llcrypt unregistered [ 3358.841769] Key type ._llcrypt registered [ 3358.844719] Key type .llcrypt registered [ 3358.971273] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3360.192872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3360.221263] alg: No test for adler32 (adler32-zlib) [ 3361.248157] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3361.569423] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3363.311158] Key type lgssc registered [ 3364.417907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3373.229305] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3373.241970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.833747] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3393.951460] LustreError: 102127:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 3420.639466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.953901] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3433.711152] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3441.358822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.679531] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3441.695973] LustreError: 103037:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3441.702194] LustreError: 103037:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 3441.713453] LustreError: 103037:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3441.828169] Lustre: server umount lustre-MDT0000 complete [ 3441.830127] LustreError: 103037:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3462.111213] Lustre: 101462:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240826/real 1781240826] req@000000002c9a1993 x1867766291238144/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781240842 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3462.140423] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.166486] LustreError: 102139:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3462.357333] Lustre: server umount lustre-OST0000 complete [ 3474.741799] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3484.763568] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3522.898879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.164418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3523.187530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3523.323436] Lustre: lustre-MDT0000: new disk, initializing [ 3523.424799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.448939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3528.033735] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3540.663253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.913248] Lustre: 105797:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3541.012528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3541.019339] Lustre: Skipped 1 previous similar message [ 3541.157845] Lustre: lustre-MDT0001: new disk, initializing [ 3541.330176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3541.364157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3541.383205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3546.755739] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3554.744523] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3559.298245] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3566.376385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.779788] Lustre: lustre-OST0000: new disk, initializing [ 3566.790044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3566.806729] Lustre: 107541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3566.954826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.374695] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3575.335883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3575.349949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3575.478082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3579.436716] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3584.125939] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3584.387551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3588.496217] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3588.733582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3591.137995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.140171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3591.169349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.269512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.278481] Lustre: Skipped 2 previous similar messages [ 3596.748538] Lustre: server umount lustre-OST0000 complete [ 3601.379330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3601.385345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3601.388983] Lustre: Skipped 2 previous similar messages [ 3601.398376] Lustre: Skipped 1 previous similar message [ 3606.499662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3606.512140] Lustre: Skipped 1 previous similar message [ 3606.938955] Lustre: server umount lustre-MDT0000 complete [ 3610.509579] LustreError: 106577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781240990 with bad export cookie 14116956012776027914 [ 3610.510397] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3610.522828] LustreError: 106577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3610.821815] Lustre: server umount lustre-MDT0001 complete [ 3618.463319] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 01:09:56 (1781240996) [ 3621.842544] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3636.024921] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3645.198368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3680.850299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.109585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3681.137283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3681.222331] Lustre: lustre-MDT0000: new disk, initializing [ 3681.278890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.292582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3685.290868] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3695.383472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.541568] Lustre: 112259:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3695.595657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3695.605318] Lustre: Skipped 1 previous similar message [ 3695.717425] Lustre: lustre-MDT0001: new disk, initializing [ 3695.909761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.964973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3695.976419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3701.379194] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3711.041142] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3715.464606] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3722.764349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3723.218617] Lustre: lustre-OST0000: new disk, initializing [ 3723.229929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3723.240644] Lustre: 113994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3723.439107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3729.563455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3729.580644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3729.726853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3729.869541] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3736.932489] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3741.739655] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3741.984678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3746.375859] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3746.688767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3750.368500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3750.370927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.377163] LustreError: Skipped 1 previous similar message [ 3750.399393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3750.407203] Lustre: Skipped 1 previous similar message [ 3760.617323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.626161] Lustre: Skipped 3 previous similar messages [ 3763.167207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3763.276349] Lustre: server umount lustre-OST0000 complete [ 3767.783921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3767.804591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.811376] Lustre: Skipped 1 previous similar message [ 3773.644398] Lustre: server umount lustre-MDT0000 complete [ 3777.634953] LustreError: 112251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241157 with bad export cookie 14116956012776028789 [ 3777.645050] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.652333] LustreError: 112251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3778.425899] Lustre: server umount lustre-MDT0001 complete [ 3788.866847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.417325] LustreError: 116069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3789.445918] LustreError: 116069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3789.513783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.915230] LustreError: 116070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.007348] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3799.012356] LustreError: 116069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.128862] LustreError: 116070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3804.336764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.977412] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3814.748784] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3819.141443] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3826.215427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.594636] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3839.239432] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3846.733948] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3855.331558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.332742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.341052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.341061] Lustre: Skipped 3 previous similar messages [ 3855.348628] LustreError: Skipped 1 previous similar message [ 3855.384318] Lustre: Skipped 2 previous similar messages [ 3856.913461] Lustre: server umount lustre-OST0000 complete [ 3866.878591] Lustre: server umount lustre-MDT0000 complete [ 3870.129405] LustreError: 116051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241250 with bad export cookie 14116956012776029678 [ 3870.134249] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3870.148333] LustreError: 116051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3870.465702] Lustre: server umount lustre-MDT0001 complete [ 3879.836764] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 3882.406108] Key type lgssc unregistered [ 3882.672319] LNet: 119342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3882.681050] LNetError: 119342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3882.705862] LNet: Removed LNI 192.168.202.159@tcp [ 3883.565922] Key type .llcrypt unregistered [ 3883.573802] Key type ._llcrypt unregistered [ 3900.251959] Key type ._llcrypt registered [ 3900.256305] Key type .llcrypt registered [ 3900.364757] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 3914.625703] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 3916.068723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3916.215859] alg: No test for adler32 (adler32-zlib) [ 3917.436685] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 3917.767224] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 3919.479192] Key type lgssc registered [ 3920.740233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3954.954238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3954.973482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.176751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3956.203912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3956.254303] Lustre: lustre-MDT0000: new disk, initializing [ 3956.311629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.329850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3960.056422] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3970.474974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.597700] Lustre: 122459:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3970.658346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3970.662316] Lustre: Skipped 1 previous similar message [ 3970.757285] Lustre: lustre-MDT0001: new disk, initializing [ 3970.891926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3970.961815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3970.979467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3974.937214] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3983.044398] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3987.257928] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3993.436409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3993.739224] Lustre: lustre-OST0000: new disk, initializing [ 3993.743803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3993.758833] Lustre: 124202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3993.879661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.318351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3999.331177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3999.436676] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 3999.454379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4006.441475] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4010.044885] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4010.125534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4013.687466] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4013.816211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4020.193414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4020.197082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.220235] LustreError: Skipped 1 previous similar message [ 4020.223823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.251878] Lustre: Skipped 1 previous similar message [ 4021.337686] Lustre: server umount lustre-OST0000 complete [ 4025.313614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.324361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4025.335364] Lustre: Skipped 1 previous similar message [ 4025.343945] Lustre: Skipped 2 previous similar messages [ 4030.432910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4030.447489] Lustre: Skipped 1 previous similar message [ 4031.428333] Lustre: server umount lustre-MDT0000 complete [ 4034.342235] LustreError: 122453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241414 with bad export cookie 3996016824000255137 [ 4034.345042] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4034.351659] LustreError: 122453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4034.663441] Lustre: server umount lustre-MDT0001 complete [ 4042.708379] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 01:17:00 (1781241420) [ 4048.961172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4049.528832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.709275] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4055.015980] LustreError: 126464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4055.030274] LustreError: 126464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4059.111659] LustreError: 126465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4062.056830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.298603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.738383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4073.233342] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4077.158357] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4079.584394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.587080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.595438] Lustre: Skipped 1 previous similar message [ 4084.705388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4084.719777] Lustre: Skipped 2 previous similar messages [ 4086.029704] Lustre: server umount lustre-MDT0000 complete [ 4087.796055] LustreError: 126459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4087.815796] LustreError: 126459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4089.592137] LustreError: 126446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241469 with bad export cookie 3996016824000256019 [ 4089.595967] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4089.601509] LustreError: 126446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4089.887767] Lustre: server umount lustre-MDT0001 complete [ 4098.712913] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 01:17:56 (1781241476) [ 4105.008333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4120.544267] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4125.663810] LustreError: 128653:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 4151.393996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.593903] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4163.184911] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4165.573854] Lustre: server umount lustre-OST0000 complete [ 4174.418703] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 01:19:11 (1781241551) [ 4180.828976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.212436] LustreError: 129984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.275520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.595919] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4186.601651] LustreError: 129985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.227282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.772966] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4204.026904] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4207.509692] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4214.065830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4214.373741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.380563] Lustre: Skipped 1 previous similar message [ 4220.446956] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4225.745386] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4237.282043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.285771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4237.306986] Lustre: Skipped 2 previous similar messages [ 4237.320989] Lustre: Skipped 2 previous similar messages [ 4243.181676] Lustre: server umount lustre-MDT0000 complete [ 4246.870053] LustreError: 129966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241626 with bad export cookie 3996016824000256684 [ 4246.880101] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4246.883852] LustreError: 129966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4247.150652] Lustre: server umount lustre-MDT0001 complete [ 4261.675669] Lustre: server umount lustre-OST0000 complete [ 4270.416975] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 01:20:47 (1781241647) [ 4277.120348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4277.686354] LustreError: 133301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.708053] LustreError: 133301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4281.825804] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4290.192735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4290.679535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4290.689603] Lustre: Skipped 1 previous similar message [ 4294.859577] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4300.505330] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4305.088430] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4313.602204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.285311] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4322.399536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4328.029863] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4334.177785] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4334.457855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4339.369897] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4339.562731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4342.753651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.763188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.777436] Lustre: Skipped 1 previous similar message [ 4342.787667] Lustre: Skipped 4 previous similar messages [ 4347.762455] Lustre: server umount lustre-OST0000 complete [ 4352.082380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.463157] Lustre: server umount lustre-MDT0000 complete [ 4362.209629] LustreError: 133302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4362.243838] LustreError: 133302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4362.451149] LustreError: 133281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241742 with bad export cookie 3996016824000257748 [ 4362.473517] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4363.038761] Lustre: server umount lustre-MDT0001 complete [ 4372.695873] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 01:22:29 (1781241749) [ 4380.560883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4396.193360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4401.311451] LustreError: 137088:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 4427.040134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.049646] Lustre: Skipped 1 previous similar message [ 4433.429569] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4438.646826] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4445.187876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.579807] LustreError: 137113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.701283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4449.156448] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4457.008126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.322043] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4466.253585] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4469.808362] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.657553] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4473.919076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4478.096699] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4478.293397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4482.021749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.034513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.041026] Lustre: Skipped 2 previous similar messages [ 4482.044362] Lustre: Skipped 4 previous similar messages [ 4486.636264] Lustre: server umount lustre-MDT0000 complete [ 4489.761907] LustreError: 137095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241869 with bad export cookie 3996016824000258777 [ 4489.765193] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4489.774313] LustreError: 137095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4490.181696] Lustre: server umount lustre-MDT0001 complete [ 4503.979564] Lustre: server umount lustre-OST0000 complete [ 4513.116686] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 01:24:50 (1781241890) [ 4519.375281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4535.009903] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4540.127702] LustreError: 140904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 4571.450687] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4576.801511] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4583.345089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.796291] LustreError: 140930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4583.810679] LustreError: 140930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4583.912933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4588.168245] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4595.752795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.716355] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4604.630775] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4608.076218] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4614.183896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.330796] Lustre: lustre-OST0001: new disk, initializing [ 4614.334856] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4614.342235] Lustre: 143436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4621.176927] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4621.424944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4621.448698] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4621.586051] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4628.859472] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4633.650985] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4633.837668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4638.009477] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4638.244313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4640.225236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4640.235967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.257750] Lustre: Skipped 1 previous similar message [ 4640.269702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4640.272583] Lustre: Skipped 2 previous similar messages [ 4642.272054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4646.391367] Lustre: server umount lustre-OST0000 complete [ 4652.515889] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4652.525437] LustreError: Skipped 1 previous similar message [ 4656.434874] Lustre: server umount lustre-OST0001 complete [ 4662.757116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4666.468396] Lustre: server umount lustre-MDT0000 complete [ 4670.944742] LustreError: 140912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242051 with bad export cookie 3996016824000259729 [ 4670.950074] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4670.953908] LustreError: 140912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4671.192302] Lustre: server umount lustre-MDT0001 complete [ 4693.876977] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 01:27:51 (1781242071) [ 4695.366165] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4697.368289] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4698.930271] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 01:27:56 (1781242076) [ 4705.213257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.537633] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4705.576697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4705.716185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.720282] Lustre: Skipped 6 previous similar messages [ 4710.134194] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4717.530532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.614863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4717.644783] Lustre: 146850:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4721.746350] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4726.677723] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4730.088496] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4735.498295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4735.648981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4741.159780] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4742.348436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4746.973408] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4751.063684] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4751.268408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4755.297778] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4755.447713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4757.472899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4757.482098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.496871] Lustre: Skipped 5 previous similar messages [ 4771.296312] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4771.551046] Lustre: server umount lustre-OST0000 complete [ 4792.806552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4798.253519] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.271162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.278524] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4798.292089] Lustre: Skipped 1 previous similar message [ 4800.673496] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4808.143523] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.842894] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4814.121107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4818.182068] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4818.345294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4822.272572] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e6ac7d20000.ost_server_uuid 50 [ 4824.046406] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e6ac7d20000.ost_server_uuid in FULL state after 0 sec [ 4824.251934] LustreError: 149055:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.59@tcp) returned error from blocking AST (req@00000000d006ae29 x1867766874489856 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006bf47992/0x3774b4212f052feb lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.59@tcp remote: 0xbdba1f69e3deee88 expref: 5 pid: 146857 timeout: 4924 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4824.282220] LustreError: lustre-MDT0000: A client on nid 192.168.202.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4824.299070] LustreError: 146846:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006bf47992/0x3774b4212f052feb lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.59@tcp remote: 0xbdba1f69e3deee88 expref: 6 pid: 146857 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4830.180409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4830.197385] LustreError: Skipped 1 previous similar message [ 4830.202125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4830.209967] Lustre: Skipped 12 previous similar messages [ 4835.972083] Lustre: server umount lustre-OST0000 complete [ 4846.040217] Lustre: server umount lustre-MDT0000 complete [ 4849.318185] LustreError: 146843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242229 with bad export cookie 3996016824000260912 [ 4849.331951] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4849.333469] LustreError: 146843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4855.819832] Lustre: server umount lustre-MDT0001 complete [ 4865.356708] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 4868.032017] Key type lgssc unregistered [ 4868.283842] LNet: 152336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4868.290972] LNetError: 152336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4868.328209] LNet: Removed LNI 192.168.202.159@tcp [ 4869.066991] Key type .llcrypt unregistered [ 4869.069537] Key type ._llcrypt unregistered [ 4895.291256] Key type ._llcrypt registered [ 4895.302106] Key type .llcrypt registered [ 4895.425860] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 4896.709947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4896.750209] alg: No test for adler32 (adler32-zlib) [ 4897.785319] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 4898.189542] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 4899.895290] Key type lgssc registered [ 4900.934084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4907.981822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4907.999979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.544439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.327798] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4919.778640] LustreError: 153742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.806295] LustreError: 153742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4922.756201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.092411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4927.646314] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4932.934770] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4937.058260] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4943.986432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4944.334644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.634537] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4954.616161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4956.236556] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4962.411833] Lustre: Failing over lustre-MDT0000 [ 4964.265547] LustreError: 154948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.321864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4964.326794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4964.599264] Lustre: server umount lustre-MDT0000 complete [ 4964.850195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.953524] LustreError: 153743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.978131] LustreError: 153743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4975.074471] LustreError: 153742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.087581] LustreError: 153742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4976.653464] LustreError: 153722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.59@tcp arrived at 1781242356 with bad export cookie 5822341186397949751 [ 4976.678422] LustreError: 153722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4980.191637] Lustre: 153051:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781242344/real 1781242344] req@00000000a723fa44 x1867767902921088/t0(0) o400->MGC192.168.202.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781242360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4980.203914] LustreError: 154453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4980.252827] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4980.307827] LustreError: 154453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4985.252497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4990.450678] LustreError: 154453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.477164] LustreError: 154453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4990.819920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4990.887307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4996.077663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4996.089316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4996.194215] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 4999.067069] Lustre: lustre-MDT0000: Denying connection for new client 68fcfeb8-2157-46a2-8fd6-de713ab43d36 (at 192.168.202.59@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5004.191474] Lustre: lustre-MDT0000: Denying connection for new client 68fcfeb8-2157-46a2-8fd6-de713ab43d36 (at 192.168.202.59@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5009.888988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5009.902621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.926561] Lustre: Skipped 1 previous similar message [ 5009.942582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5011.427584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.574140] Lustre: server umount lustre-OST0000 complete [ 5018.620355] LustreError: 157117:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5018.629914] Lustre: 156482:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5018.652737] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5018.678697] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5018.712878] LustreError: 156482:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000013ec514d x1867767902941184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5018.748480] LustreError: 156482:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5018.760937] LustreError: 156482:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5018.772145] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5019.033903] Lustre: server umount lustre-MDT0000 complete [ 5021.681812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.721371] Lustre: Skipped 1 previous similar message [ 5021.738852] LustreError: 153739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.912806] LustreError: 155358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242403 with bad export cookie 5822341186397949849 [ 5022.921697] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.924431] LustreError: 155358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5022.935593] LustreError: 153050:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5025.308794] Lustre: server umount lustre-MDT0001 complete [ 5037.955179] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5040.803453] Key type lgssc unregistered [ 5041.143592] LNet: 157852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5041.153640] LNetError: 157852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5041.175859] LNet: Removed LNI 192.168.202.159@tcp [ 5041.938904] Key type .llcrypt unregistered [ 5041.941761] Key type ._llcrypt unregistered [ 5066.016609] Key type ._llcrypt registered [ 5066.018857] Key type .llcrypt registered [ 5066.169817] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5067.784427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5067.857505] alg: No test for adler32 (adler32-zlib) [ 5068.955502] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5069.213850] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5070.873104] Key type lgssc registered [ 5071.994655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5079.553349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5079.581390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.163965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5085.330392] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5091.298986] LustreError: 159249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.329149] LustreError: 159249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5093.614775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.883540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5097.856531] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5104.195622] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5107.928459] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5114.757377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5115.112692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.787229] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5126.256474] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5126.403465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5133.793562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5133.805728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.827443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5136.868178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.871577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5136.894202] Lustre: Skipped 1 previous similar message [ 5139.895875] Lustre: server umount lustre-OST0000 complete [ 5145.057717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5145.069246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.100514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.106980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5147.109629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.849520] Lustre: server umount lustre-MDT0000 complete [ 5152.225093] LustreError: 159245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.257032] LustreError: 159245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5153.464535] LustreError: 159229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242533 with bad export cookie 15690853598040265535 [ 5153.484656] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.489177] LustreError: 159229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.817897] Lustre: server umount lustre-MDT0001 complete [ 5164.994383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5168.013664] Key type lgssc unregistered [ 5168.412309] LNet: 162425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5168.419973] LNetError: 162425:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5168.450786] LNet: Removed LNI 192.168.202.159@tcp [ 5169.358424] Key type .llcrypt unregistered [ 5169.362164] Key type ._llcrypt unregistered [ 5204.656730] Key type ._llcrypt registered [ 5204.659115] Key type .llcrypt registered [ 5204.779326] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5205.732842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5205.896529] alg: No test for adler32 (adler32-zlib) [ 5206.950919] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5207.167927] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5208.857294] Key type lgssc registered [ 5209.977361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5216.846540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5216.869102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.308770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.284941] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5228.522321] LustreError: 164111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.534505] LustreError: 164111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5230.288759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5230.822408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5235.834464] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5242.904733] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5246.188322] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5252.935190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5253.295312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.160714] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5263.548052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5264.097158] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5271.009223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5271.024347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.038947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.082593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.085244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.095182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.200890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5279.212583] Lustre: Skipped 2 previous similar messages [ 5284.319502] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5284.325716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.358713] Lustre: Skipped 1 previous similar message [ 5284.481368] Lustre: server umount lustre-OST0000 complete [ 5289.441641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5289.464192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5289.467827] Lustre: Skipped 1 previous similar message [ 5289.492228] Lustre: Skipped 1 previous similar message [ 5299.688312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5299.699768] Lustre: Skipped 3 previous similar messages [ 5303.263129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5303.433601] Lustre: server umount lustre-MDT0000 complete [ 5304.803224] LustreError: 165270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.816721] LustreError: 165270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5306.822816] LustreError: 166554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242686 with bad export cookie 5691914948713001367 [ 5306.836170] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5307.089075] Lustre: server umount lustre-MDT0001 complete [ 5319.650215] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5322.091394] Key type lgssc unregistered [ 5322.338469] LNet: 167290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5322.350318] LNetError: 167290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5322.389136] LNet: Removed LNI 192.168.202.159@tcp [ 5323.378656] Key type .llcrypt unregistered [ 5323.386363] Key type ._llcrypt unregistered [ 5345.929852] Key type ._llcrypt registered [ 5345.933223] Key type .llcrypt registered [ 5346.075605] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5347.252751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5347.354325] alg: No test for adler32 (adler32-zlib) [ 5348.474327] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5348.725943] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5350.472977] Key type lgssc registered [ 5351.689722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5362.835262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5362.849156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5364.180529] Lustre: *** cfs_fail_loc=135, val=0*** [ 5364.240968] LustreError: 168797:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5364.247628] LustreError: 168797:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.159@tcp: cfg command failed: rc = -2 [ 5364.256721] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5364.256721] [ 5364.263631] LustreError: MGC192.168.202.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5364.275508] LustreError: 168758:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5364.283646] LustreError: 168758:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5364.289598] LustreError: 168758:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5364.296481] LustreError: 168758:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5364.357297] Lustre: server umount lustre-MDT0000 complete [ 5364.361114] LustreError: 168758:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5370.336408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.910288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.676922] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5380.068237] LustreError: 169169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5380.083046] LustreError: 169169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5382.182091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.557508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.665614] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5390.520550] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5394.007193] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5397.986958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5397.994452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.010826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5400.036130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5400.043787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5405.155857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5405.163246] Lustre: Skipped 2 previous similar messages [ 5407.204192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5407.211352] Lustre: Skipped 1 previous similar message [ 5410.271191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5410.448329] Lustre: server umount lustre-MDT0000 complete [ 5412.323495] LustreError: 169165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.369163] LustreError: 169165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5414.348706] LustreError: 169149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242794 with bad export cookie 7636517417748998147 [ 5414.359571] LustreError: 169149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5414.361255] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5414.718702] Lustre: server umount lustre-MDT0001 complete [ 5424.520961] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5427.093564] Key type lgssc unregistered [ 5427.431617] LNet: 171247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5427.439625] LNetError: 171247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5427.462565] LNet: Removed LNI 192.168.202.159@tcp [ 5428.289117] Key type .llcrypt unregistered [ 5428.291050] Key type ._llcrypt unregistered [ 5448.315916] Key type ._llcrypt registered [ 5448.318684] Key type .llcrypt registered [ 5448.410992] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5448.613374] Key type .llcrypt unregistered [ 5448.615628] Key type ._llcrypt unregistered [ 5463.709920] Key type ._llcrypt registered [ 5463.715573] Key type .llcrypt registered [ 5463.823615] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5464.689648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5464.857500] alg: No test for adler32 (adler32-zlib) [ 5465.897962] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5466.088569] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5467.747136] Key type lgssc registered [ 5468.646053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5475.447492] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5475.462961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.039450] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5496.160253] LustreError: 173337:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.159@tcp: failed processing log, type 4: rc = -110 [ 5522.847550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.543862] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5534.226396] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5539.688227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.081468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.142933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5543.660465] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5545.451407] LustreError: 173363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.484824] LustreError: 173363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5550.569891] LustreError: 173362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.789994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5552.041690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.649305] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5563.093530] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5566.881879] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5569.272668] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5581.280977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5581.286155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.300604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.817463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.822491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.851724] Lustre: Skipped 1 previous similar message [ 5586.953626] Lustre: server umount lustre-OST0000 complete [ 5593.059073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.061948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5593.066366] Lustre: Skipped 1 previous similar message [ 5593.074413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5596.289861] Lustre: server umount lustre-MDT0000 complete [ 5598.189418] LustreError: 174265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5598.219461] LustreError: 174265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5599.251790] LustreError: 173345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781242979 with bad export cookie 1744039287822477447 [ 5599.255581] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5599.258791] LustreError: 173345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5599.484940] Lustre: server umount lustre-MDT0001 complete [ 5607.578759] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5610.080659] Key type lgssc unregistered [ 5610.443672] LNet: 176992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5610.455624] LNetError: 176992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5610.467812] LNet: Removed LNI 192.168.202.159@tcp [ 5611.281137] Key type .llcrypt unregistered [ 5611.283429] Key type ._llcrypt unregistered [ 5632.726777] Key type ._llcrypt registered [ 5632.728700] Key type .llcrypt registered [ 5632.810960] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5633.891842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5633.915745] alg: No test for adler32 (adler32-zlib) [ 5634.962915] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5635.147221] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5636.788192] Key type lgssc registered [ 5637.819633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5643.901273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5643.929515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.271513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.000771] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5655.523025] LustreError: 178380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.545447] LustreError: 178380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5656.249118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.513536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5660.262873] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5665.627828] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5669.360098] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5675.838278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5676.108092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.271852] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5685.496712] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5690.486338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5695.103828] Lustre: Failing over lustre-MDT0000 [ 5695.378795] Lustre: server umount lustre-MDT0000 complete [ 5695.971847] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.973149] LustreError: 178381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.982408] Lustre: Skipped 1 previous similar message [ 5696.001234] LustreError: 178381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5697.436839] LustreError: 178375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.094694] LustreError: 179546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.115438] LustreError: 179546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5706.210853] LustreError: 178380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.232900] LustreError: 178380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5712.521571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.634632] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5712.821279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.850570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5716.904991] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5717.892087] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5718.002592] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5718.031211] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5718.061910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5720.230553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5723.171293] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5732.832943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5732.845086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5732.856152] Lustre: Skipped 1 previous similar message [ 5732.860498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.177378] Lustre: server umount lustre-OST0000 complete [ 5743.587486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.593882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5743.604362] Lustre: Skipped 2 previous similar messages [ 5743.606066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5743.625078] Lustre: Skipped 3 previous similar messages [ 5748.704053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.710442] Lustre: Skipped 1 previous similar message [ 5753.827175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.335671] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5754.527486] Lustre: server umount lustre-MDT0000 complete [ 5757.393513] LustreError: 181771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243137 with bad export cookie 11943275946158057432 [ 5757.394729] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5757.401682] LustreError: 181771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5757.615357] Lustre: server umount lustre-MDT0001 complete [ 5765.192291] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5766.999523] Key type lgssc unregistered [ 5767.252326] LNet: 182800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5767.258974] LNetError: 182800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5768.293715] LNet: Removed LNI 192.168.202.159@tcp [ 5768.971515] Key type .llcrypt unregistered [ 5768.973892] Key type ._llcrypt unregistered [ 5787.286942] Key type ._llcrypt registered [ 5787.289135] Key type .llcrypt registered [ 5787.359806] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5788.187947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5788.204902] alg: No test for adler32 (adler32-zlib) [ 5789.284352] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5789.479416] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5791.136107] Key type lgssc registered [ 5792.078347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5798.269799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5798.286370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.723435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.230383] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5809.130364] LustreError: 184179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.147615] LustreError: 184179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5810.205679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.446579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5813.817555] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5819.124307] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5822.342865] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5827.342142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5827.594248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.745275] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5835.361496] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5844.012369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5848.150565] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5853.225910] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5865.654866] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5874.657562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.659194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.664345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.671602] Lustre: Skipped 1 previous similar message [ 5879.780877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5879.782674] Lustre: Skipped 2 previous similar messages [ 5880.613762] Lustre: server umount lustre-OST0000 complete [ 5884.896311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.900632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5884.905646] Lustre: Skipped 1 previous similar message [ 5884.914993] Lustre: Skipped 1 previous similar message [ 5890.016974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5890.026443] Lustre: Skipped 1 previous similar message [ 5895.137860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5895.145398] Lustre: Skipped 1 previous similar message [ 5897.183102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5897.284495] Lustre: server umount lustre-MDT0000 complete [ 5899.674770] LustreError: 184160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243279 with bad export cookie 3998421456894718578 [ 5899.680782] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5899.684836] LustreError: 184160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5899.865337] Lustre: server umount lustre-MDT0001 complete [ 5906.732710] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 5908.573983] Key type lgssc unregistered [ 5908.729795] LNet: 187688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5908.734791] LNetError: 187688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5908.747255] LNet: Removed LNI 192.168.202.159@tcp [ 5909.339665] Key type .llcrypt unregistered [ 5909.341182] Key type ._llcrypt unregistered [ 5921.959401] Key type ._llcrypt registered [ 5921.961181] Key type .llcrypt registered [ 5922.029242] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 5922.673167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5922.727852] alg: No test for adler32 (adler32-zlib) [ 5923.762117] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 5923.938660] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 5925.575293] Key type lgssc registered [ 5926.397551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5931.328433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5931.338913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.699508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.514238] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5941.346199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5941.662084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.521812] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5948.615305] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5951.505337] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5956.048711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.361778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.427303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5961.581607] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 5965.359713] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5969.224424] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5978.457313] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5986.630515] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5990.682702] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6002.668175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6002.675777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6002.676099] Lustre: Skipped 1 previous similar message [ 6002.683938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6002.709352] Lustre: Skipped 1 previous similar message [ 6005.747132] Lustre: server umount lustre-OST0000 complete [ 6012.897342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6012.900627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6012.912102] Lustre: Skipped 1 previous similar message [ 6012.933124] Lustre: Skipped 1 previous similar message [ 6015.453113] Lustre: server umount lustre-MDT0000 complete [ 6018.016275] LustreError: 188916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.030925] LustreError: 188916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6018.375925] LustreError: 188897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243398 with bad export cookie 1175802486376936722 [ 6018.379941] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6018.383701] LustreError: 188897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6018.609243] Lustre: server umount lustre-MDT0001 complete [ 6026.587262] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6028.624250] Key type lgssc unregistered [ 6028.886093] LNet: 193530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6028.894605] LNetError: 193530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6028.909915] LNet: Removed LNI 192.168.202.159@tcp [ 6029.654170] Key type .llcrypt unregistered [ 6029.656467] Key type ._llcrypt unregistered [ 6047.475073] Key type ._llcrypt registered [ 6047.478699] Key type .llcrypt registered [ 6047.574309] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6048.451148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.515674] alg: No test for adler32 (adler32-zlib) [ 6049.513215] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6049.681587] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6051.375371] Key type lgssc registered [ 6052.117584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6057.410702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6057.420785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.720740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.721185] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6068.195681] LustreError: 194904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.219343] LustreError: 194904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6068.251052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.526471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6071.657178] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6076.216425] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6079.227501] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6084.268727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6084.570359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6088.907400] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6092.724722] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6103.036924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6113.248322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6113.255310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.256938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6113.279511] Lustre: Skipped 1 previous similar message [ 6118.375568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6118.387829] Lustre: Skipped 2 previous similar messages [ 6118.916757] Lustre: server umount lustre-OST0000 complete [ 6123.489768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.494574] Lustre: Skipped 1 previous similar message [ 6123.497300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.501141] Lustre: Skipped 1 previous similar message [ 6128.608517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6128.610964] Lustre: Skipped 1 previous similar message [ 6133.729602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6133.731373] Lustre: Skipped 1 previous similar message [ 6136.287144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6136.419981] Lustre: server umount lustre-MDT0000 complete [ 6138.855193] LustreError: 197131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6138.866584] LustreError: 197131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6138.989166] LustreError: 194886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243519 with bad export cookie 82707987094048506 [ 6138.991940] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.008802] LustreError: 194886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6139.372807] Lustre: server umount lustre-MDT0001 complete [ 6147.953801] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6149.939247] Key type lgssc unregistered [ 6150.151930] LNet: 198252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6150.156312] LNetError: 198252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6151.204775] LNet: Removed LNI 192.168.202.159@tcp [ 6151.801266] Key type .llcrypt unregistered [ 6151.804225] Key type ._llcrypt unregistered [ 6166.183448] Key type ._llcrypt registered [ 6166.186097] Key type .llcrypt registered [ 6166.240701] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6166.996263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6167.050438] alg: No test for adler32 (adler32-zlib) [ 6168.061545] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6168.233438] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6169.839240] Key type lgssc registered [ 6170.771764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6175.834924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6175.868085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.184234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.503497] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6186.773867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6187.362483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.931197] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6194.941307] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6197.623326] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6202.248288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6202.487343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.507863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6208.083580] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6212.808397] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6223.328975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.330660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.338629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6223.348145] Lustre: Skipped 1 previous similar message [ 6223.361206] Lustre: Skipped 1 previous similar message [ 6228.458437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6228.463358] Lustre: Skipped 1 previous similar message [ 6228.644720] Lustre: server umount lustre-OST0000 complete [ 6233.571213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6233.572781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.586723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6233.596395] Lustre: Skipped 1 previous similar message [ 6237.573166] Lustre: server umount lustre-MDT0000 complete [ 6238.698148] LustreError: 200189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6238.712514] LustreError: 200189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6240.814078] LustreError: 199459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243620 with bad export cookie 17843083659369922933 [ 6240.820720] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.823930] LustreError: 199459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6241.171272] Lustre: server umount lustre-MDT0001 complete [ 6248.500168] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6250.560555] Key type lgssc unregistered [ 6250.816678] LNet: 202804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6250.822756] LNetError: 202804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6250.856926] LNet: Removed LNI 192.168.202.159@tcp [ 6251.442604] Key type .llcrypt unregistered [ 6251.444806] Key type ._llcrypt unregistered [ 6263.816522] Key type ._llcrypt registered [ 6263.819469] Key type .llcrypt registered [ 6263.915426] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6264.596950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6264.704716] alg: No test for adler32 (adler32-zlib) [ 6265.669928] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6265.834204] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6267.479220] Key type lgssc registered [ 6268.382333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6273.441498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6273.451163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.800618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6277.383669] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6283.471388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.680121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.527047] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6290.064547] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6292.311300] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6296.859696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6297.111698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6299.190526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6300.621046] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6304.531674] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6311.943958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6311.983957] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6312.126823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6316.393599] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6320.409272] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6321.597513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6333.105868] Lustre: Permanently deactivating lustre-OST0001 [ 6333.165832] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6339.183736] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6346.959935] Lustre: Permanently reactivating lustre-OST0001 [ 6346.976132] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6346.978633] Lustre: Skipped 2 previous similar messages [ 6355.172054] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.209103] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6355.216397] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6355.232601] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6355.329930] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6357.486692] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.497892] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6357.506751] Lustre: Skipped 2 previous similar messages [ 6357.528274] Lustre: Skipped 1 previous similar message [ 6362.599295] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6362.651247] Lustre: Skipped 1 previous similar message [ 6363.534532] Lustre: server umount lustre-OST0001 complete [ 6367.718287] LustreError: 205626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.771651] LustreError: 205626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6372.834789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.837089] LustreError: 205625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.880737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.924835] LustreError: 205625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6372.965450] Lustre: Skipped 1 previous similar message [ 6377.954130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6377.970170] LustreError: 205625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.971891] Lustre: Skipped 1 previous similar message [ 6378.025085] LustreError: 205625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6379.069359] Lustre: server umount lustre-OST0000 complete [ 6386.143934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6386.146535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.162985] Lustre: Skipped 1 previous similar message [ 6386.171276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.221978] Lustre: server umount lustre-MDT0000 complete [ 6393.329404] LustreError: 204026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.359218] LustreError: 204026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6397.940713] LustreError: 206529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781243778 with bad export cookie 271369222676980374 [ 6397.983715] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6397.984749] LustreError: 206529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6398.665642] Lustre: server umount lustre-MDT0001 complete [ 6414.728318] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6417.937262] Key type lgssc unregistered [ 6418.397826] LNet: 208612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6418.409927] LNetError: 208612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6418.439540] LNet: Removed LNI 192.168.202.159@tcp [ 6419.702648] Key type .llcrypt unregistered [ 6419.706073] Key type ._llcrypt unregistered [ 6460.437411] Key type ._llcrypt registered [ 6460.443907] Key type .llcrypt registered [ 6460.528314] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6461.789062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6461.847063] alg: No test for adler32 (adler32-zlib) [ 6462.878560] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6463.068341] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6464.799272] Key type lgssc registered [ 6466.001553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6474.259677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6474.285345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.589249] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6475.627658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6475.840784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.240621] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6488.426441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.530573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6488.561418] Lustre: 210801:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6488.723118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6492.884202] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6499.052641] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6503.125652] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6509.635928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6509.855594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6510.035298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.031300] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6517.902907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6521.832490] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6525.553713] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6532.056639] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6541.716288] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6552.051978] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6563.218808] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6563.228540] Lustre: Skipped 1 previous similar message [ 6579.837508] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6579.843416] Lustre: Skipped 1 previous similar message [ 6619.125992] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6619.130831] Lustre: Skipped 3 previous similar messages [ 6690.390534] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6690.406028] Lustre: Skipped 7 previous similar messages [ 6721.880526] Lustre: server umount lustre-OST0000 complete [ 6728.170623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.186076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6728.186127] Lustre: Skipped 1 previous similar message [ 6728.229441] Lustre: Skipped 1 previous similar message [ 6733.280606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.306500] Lustre: Skipped 1 previous similar message [ 6736.469876] Lustre: server umount lustre-MDT0000 complete [ 6738.399140] Lustre: 210124:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244102/real 1781244102] req@0000000020ee415e x1867769544112128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781244118 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6738.465549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.423181] Lustre: 210122:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244103/real 1781244103] req@00000000abb47123 x1867769544113024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781244119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6742.608568] LustreError: 210793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244122 with bad export cookie 1753837963075119658 [ 6742.610342] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6742.616153] LustreError: 210793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6743.373900] Lustre: server umount lustre-MDT0001 complete [ 6758.254233] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6761.231699] Key type lgssc unregistered [ 6761.503491] LNet: 215274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6761.508285] LNetError: 215274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6761.529674] LNet: Removed LNI 192.168.202.159@tcp [ 6762.369732] Key type .llcrypt unregistered [ 6762.371897] Key type ._llcrypt unregistered [ 6786.785816] Key type ._llcrypt registered [ 6786.791792] Key type .llcrypt registered [ 6786.930816] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6788.354027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6788.410412] alg: No test for adler32 (adler32-zlib) [ 6789.531277] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6789.822550] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6791.495749] Key type lgssc registered [ 6792.949924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6802.294522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6802.314778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.768664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6809.002152] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6813.156159] LustreError: 216672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6813.188554] LustreError: 216672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6818.280375] LustreError: 216673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.459508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.967344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6823.373368] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6829.335822] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6832.913152] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6840.128750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6840.639284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6846.680187] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_default_debug -1 all [ 6848.822945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6852.780521] Lustre: DEBUG MARKER: oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6874.493361] Lustre: server umount lustre-OST0000 complete [ 6879.714161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.721473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.752666] Lustre: Skipped 1 previous similar message [ 6884.835117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6884.859661] Lustre: Skipped 2 previous similar messages [ 6889.076943] Lustre: server umount lustre-MDT0000 complete [ 6889.951625] Lustre: 215985:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244254/real 1781244254] req@00000000054fc2ea x1867769885789312/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781244270 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6889.964788] LustreError: 217671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6890.000882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6890.054900] LustreError: 217671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6893.516951] LustreError: 216653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244273 with bad export cookie 15014624954300402984 [ 6893.521733] LustreError: MGC192.168.202.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6893.529490] LustreError: 216653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6893.971578] Lustre: server umount lustre-MDT0001 complete [ 6906.111957] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 6909.654527] Key type lgssc unregistered [ 6910.012842] LNet: 220005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6910.022617] LNetError: 220005:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6910.058359] LNet: Removed LNI 192.168.202.159@tcp [ 6911.042153] Key type .llcrypt unregistered [ 6911.044111] Key type ._llcrypt unregistered [ 6937.711991] Key type ._llcrypt registered [ 6937.724491] Key type .llcrypt registered [ 6937.865590] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 6939.302440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6939.359880] alg: No test for adler32 (adler32-zlib) [ 6940.522961] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 6940.772795] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 6942.432397] Key type lgssc registered [ 6943.604233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7213.405978] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7216.249910] Key type lgssc unregistered [ 7216.501171] LNet: 221999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7216.516743] LNetError: 221999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7216.528967] LNet: Removed LNI 192.168.202.159@tcp [ 7217.278569] Key type .llcrypt unregistered [ 7217.281783] Key type ._llcrypt unregistered [ 7245.126141] Key type ._llcrypt registered [ 7245.128322] Key type .llcrypt registered [ 7245.231988] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing set_hostid [ 7261.467539] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing load_modules_local [ 7262.624857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7262.882968] alg: No test for adler32 (adler32-zlib) [ 7264.055810] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 7264.294709] LNet: Added LNI 192.168.202.159@tcp [8/256/0/180] [ 7266.023195] Key type lgssc registered [ 7267.076130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7300.886913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6985 sec ========== 02:11:18 (1781244678) [ 7302.994910] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:11:20 (1781244680) === [ 7307.174183] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:11:24 (1781244684) === [ 7348.711819] Lustre: DEBUG MARKER: oleg259-server.virtnet: executing unload_modules_local [ 7351.504254] Key type lgssc unregistered [ 7351.835641] LNet: 227188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7351.843632] LNetError: 227188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7351.858781] LNet: Removed LNI 192.168.202.159@tcp [ 7352.830543] Key type .llcrypt unregistered [ 7352.833730] Key type ._llcrypt unregistered