[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486356947 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002237] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007619] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011093] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013073] *** VALIDATE selinux *** [ 0.020636] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025554] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] *** VALIDATE tmpfs *** [ 0.029470] *** VALIDATE proc *** [ 0.031169] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033285] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034198] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036038] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039976] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.041201] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042631] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043043] ... version: 2 [ 0.044012] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046015] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050324] rcu: Hierarchical SRCU implementation. [ 0.052488] smp: Bringing up secondary CPUs ... [ 0.053629] x86: Booting SMP configuration: [ 0.054028] .... node #0, CPUs: #1 #2 #3 [ 0.058018] smp: Brought up 1 node, 4 CPUs [ 0.059927] smpboot: Max logical packages: 1 [ 0.060017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.138396] node 0 deferred pages initialised in 75ms [ 0.142169] devtmpfs: initialized [ 0.143194] x86/mm: Memory block size: 128MB [ 0.145335] gcov: version magic: 0x41383552 [ 0.147311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149290] pinctrl core: initialized pinctrl subsystem [ 0.150270] [ 0.150734] ************************************************************* [ 0.151012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152015] ** ** [ 0.153015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154016] ** ** [ 0.155013] ** This means that this kernel is built to expose internal ** [ 0.156013] ** IOMMU data structures, which may compromise security on ** [ 0.157010] ** your system. ** [ 0.158015] ** ** [ 0.159013] ** If you see this message and you are not debugging the ** [ 0.160011] ** kernel, report this immediately to your vendor! ** [ 0.161012] ** ** [ 0.162012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163009] ************************************************************* [ 0.164738] NET: Registered protocol family 16 [ 0.165456] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169049] cpuidle: using governor menu [ 0.172043] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175519] PCI: Using configuration type 1 for base access [ 0.178130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191221] cryptd: max_cpu_qlen set to 1000 [ 0.193280] ACPI: Added _OSI(Module Device) [ 0.194015] ACPI: Added _OSI(Processor Device) [ 0.195015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200378] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203393] ACPI: Interpreter enabled [ 0.204054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205013] ACPI: Using IOAPIC for interrupt routing [ 0.206153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207381] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217166] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.219018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222339] acpiphp: Slot [2] registered [ 0.223164] acpiphp: Slot [5] registered [ 0.224126] acpiphp: Slot [6] registered [ 0.225140] acpiphp: Slot [7] registered [ 0.226098] acpiphp: Slot [8] registered [ 0.227109] acpiphp: Slot [9] registered [ 0.228143] acpiphp: Slot [10] registered [ 0.229134] acpiphp: Slot [3] registered [ 0.230105] acpiphp: Slot [4] registered [ 0.231095] acpiphp: Slot [11] registered [ 0.232102] acpiphp: Slot [12] registered [ 0.233100] acpiphp: Slot [13] registered [ 0.234104] acpiphp: Slot [14] registered [ 0.235107] acpiphp: Slot [15] registered [ 0.236114] acpiphp: Slot [16] registered [ 0.237075] acpiphp: Slot [17] registered [ 0.239140] acpiphp: Slot [18] registered [ 0.241125] acpiphp: Slot [19] registered [ 0.244140] acpiphp: Slot [20] registered [ 0.246203] acpiphp: Slot [21] registered [ 0.247153] acpiphp: Slot [22] registered [ 0.249143] acpiphp: Slot [23] registered [ 0.251155] acpiphp: Slot [24] registered [ 0.252144] acpiphp: Slot [25] registered [ 0.254165] acpiphp: Slot [26] registered [ 0.255184] acpiphp: Slot [27] registered [ 0.257148] acpiphp: Slot [28] registered [ 0.258137] acpiphp: Slot [29] registered [ 0.260164] acpiphp: Slot [30] registered [ 0.262197] acpiphp: Slot [31] registered [ 0.264126] PCI host bridge to bus 0000:00 [ 0.265030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.270034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281236] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.284976] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.288462] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.301776] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.305922] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.315559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318774] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.321071] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.325311] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332031] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.345023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.364025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395072] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.439022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453079] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.458019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.463016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.489513] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.498015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.503959] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.525018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.537088] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.546028] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.557027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.582024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597183] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.614018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.634019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.648399] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.650446] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.651540] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.655458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.658285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662223] iommu: Default domain type: Passthrough [ 0.664635] SCSI subsystem initialized [ 0.666141] ACPI: bus type USB registered [ 0.668168] usbcore: registered new interface driver usbfs [ 0.670083] usbcore: registered new interface driver hub [ 0.673098] usbcore: registered new device driver usb [ 0.675207] pps_core: LinuxPPS API ver. 1 registered [ 0.677015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.680099] PTP clock support registered [ 0.683047] EDAC MC: Ver: 3.0.0 [ 0.685130] PCI: Using ACPI for IRQ routing [ 0.686550] NetLabel: Initializing [ 0.688016] NetLabel: domain hash size = 128 [ 0.689007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.690073] NetLabel: unlabeled traffic allowed by default [ 0.693152] vgaarb: loaded [ 0.694223] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.696027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.703436] clocksource: Switched to clocksource kvm-clock [ 0.817181] VFS: Disk quotas dquot_6.6.0 [ 0.818719] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821396] *** VALIDATE ramfs *** [ 0.822865] *** VALIDATE hugetlbfs *** [ 0.824119] pnp: PnP ACPI init [ 0.827493] pnp: PnP ACPI: found 6 devices [ 0.865789] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869087] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871033] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.872914] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875302] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.877869] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.881266] NET: Registered protocol family 2 [ 0.883694] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.892567] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.896289] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.901637] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.904777] TCP: Hash tables configured (established 65536 bind 65536) [ 0.907900] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.911366] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914202] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917344] NET: Registered protocol family 1 [ 0.920980] RPC: Registered named UNIX socket transport module. [ 0.923233] RPC: Registered udp transport module. [ 0.924971] RPC: Registered tcp transport module. [ 0.926620] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.928939] NET: Registered protocol family 44 [ 0.930399] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.932276] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.934350] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.936866] PCI: CLS 0 bytes, default 64 [ 0.938898] Unpacking initramfs... [ 2.328445] debug: unmapping init [mem 0xffff93f23cc54000-0xffff93f23ffbffff] [ 2.332906] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.335026] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.337867] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.837529] Initialise system trusted keyrings [ 2.839458] Key type blacklist registered [ 2.842094] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.852512] zbud: loaded [ 2.855835] *** VALIDATE nfs *** [ 2.857191] *** VALIDATE nfs4 *** [ 2.858877] pstore: using deflate compression [ 2.862846] Platform Keyring initialized [ 2.970845] NET: Registered protocol family 38 [ 2.972730] Key type asymmetric registered [ 2.975896] Asymmetric key parser 'x509' registered [ 2.979840] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.983803] io scheduler mq-deadline registered [ 2.985408] io scheduler kyber registered [ 2.987236] io scheduler bfq registered [ 2.989433] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.992733] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.995694] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.998780] ACPI: Power Button [PWRF] [ 3.003520] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.011266] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.025355] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.037032] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.055431] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.082068] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.112989] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.118457] Non-volatile memory driver v1.3 [ 3.120053] Linux agpgart interface v0.103 [ 3.157407] virtio_blk virtio1: [vda] 145176 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.161816] vda: detected capacity change from 0 to 74330112 [ 3.179431] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.182792] vdb: detected capacity change from 0 to 1073741824 [ 3.198638] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.201547] vdc: detected capacity change from 0 to 2621440000 [ 3.217256] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.220442] vdd: detected capacity change from 0 to 2621440000 [ 3.237145] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.240284] vde: detected capacity change from 0 to 4294967296 [ 3.255955] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.259434] vdf: detected capacity change from 0 to 4294967296 [ 3.271418] libphy: Fixed MDIO Bus: probed [ 3.278029] usbcore: registered new interface driver usbserial_generic [ 3.280585] usbserial: USB Serial support registered for generic [ 3.283150] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.287508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.289389] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.291662] mousedev: PS/2 mouse device common for all mice [ 3.297549] rtc_cmos 00:05: RTC can wake from S4 [ 3.297634] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.301804] rtc_cmos 00:05: registered as rtc0 [ 3.306781] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.306892] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.310134] intel_pstate: CPU model not supported [ 3.316820] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.320936] hid: raw HID events driver (C) Jiri Kosina [ 3.325787] usbcore: registered new interface driver usbhid [ 3.330905] usbhid: USB HID core driver [ 3.332579] drop_monitor: Initializing network drop monitor service [ 3.335139] Initializing XFRM netlink socket [ 3.338234] NET: Registered protocol family 10 [ 3.343207] Segment Routing with IPv6 [ 3.344563] NET: Registered protocol family 17 [ 3.346636] mpls_gso: MPLS GSO support [ 3.352808] RAS: Correctable Errors collector initialized. [ 3.357172] AVX version of gcm_enc/dec engaged. [ 3.360424] AES CTR mode by8 optimization enabled [ 3.444431] sched_clock: Marking stable (3444324437, 0)->(4407415794, -963091357) [ 3.448822] registered taskstats version 1 [ 3.454184] Loading compiled-in X.509 certificates [ 3.457257] zswap: loaded using pool lzo/zbud [ 3.483284] Key type big_key registered [ 3.495704] Key type encrypted registered [ 3.497585] ima: No TPM chip found, activating TPM-bypass! [ 3.501784] ima: Allocated hash algorithm: sha1 [ 3.503661] ima: No architecture policies found [ 3.505601] evm: Initialising EVM extended attributes: [ 3.507688] evm: security.selinux [ 3.509089] evm: security.ima [ 3.510350] evm: security.capability [ 3.511829] evm: HMAC attrs: 0x1 [ 3.514354] rtc_cmos 00:05: setting system clock to 2026-06-23 13:52:52 UTC (1782222772) [ 3.522274] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 3.525683] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 3.539103] Write protecting the kernel read-only data: 28672k [ 3.544724] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.552554] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 3.598810] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.608557] systemd[1]: Detected virtualization kvm. [ 3.610628] systemd[1]: Detected architecture x86-64. [ 3.612549] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.637098] systemd[1]: No hostname configured. [ 3.639098] systemd[1]: Set hostname to . [ 3.640974] random: systemd: uninitialized urandom read (16 bytes read) [ 3.643337] systemd[1]: Initializing machine ID from random generator. [ 3.688672] random: ln: uninitialized urandom read (6 bytes read) [ 3.781298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.786483] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.792269] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.797316] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.554917] device-mapper: uevent: version 1.0.3 [ 4.557531] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 4.895354] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.350391] virtio_net virtio0 ens2: renamed from eth0 [ 5.417325] scsi host0: ata_piix [ 5.433278] scsi host1: ata_piix [ 5.437289] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.440229] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.154895] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.017171] random: crng init done [ 10.018718] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.501448] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.617650] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.883829] SELinux: Disabled at runtime. [ 11.947436] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.956234] systemd[1]: Detected virtualization kvm. [ 11.957952] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.464304] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.467779] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.473535] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.477284] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.479916] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.488376] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.497558] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required[ 12.520107] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.914862] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.274269] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.342965] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.460173] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.471635] EDAC sbridge: Ver: 1.1.2 [ 15.067719] Key type dns_resolver registered [ 15.378548] NFS: Registering the id_resolver key type [ 15.380351] Key type id_resolver registered [ 15.381534] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 70.517289] libcfs: loading out-of-tree module taints kernel. [ 70.555103] Key type ._llcrypt registered [ 70.559199] Key type .llcrypt registered [ 70.686172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 89.366665] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 90.889882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.901883] alg: No test for adler32 (adler32-zlib) [ 92.367302] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 93.208346] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 94.983575] Key type lgssc registered [ 96.685119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.193086] hrtimer: interrupt took 2079045 ns [ 113.923375] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.949248] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 171.001086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.017604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.305745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.355375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.430347] Lustre: lustre-MDT0000: new disk, initializing [ 172.522716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.547891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.721856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.382183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.479474] Lustre: 6509:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.508555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.513032] Lustre: Skipped 1 previous similar message [ 193.576747] Lustre: lustre-MDT0001: new disk, initializing [ 193.639594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.665578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.684892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.648480] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.109650] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.490309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.767913] Lustre: lustre-OST0000: new disk, initializing [ 215.773228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.780765] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.825952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.725330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.738824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.854677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.599853] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.402482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.515724] Lustre: lustre-OST0001: new disk, initializing [ 237.518771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.522738] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.574705] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.235364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.254513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.327386] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.899546] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 254.810458] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 262.769589] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 269.016363] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 273.269044] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 277.874720] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 280.096700] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 282.371967] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 283.858141] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 09:57:30 EDT 2026 [ 299.722874] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 300.980365] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.385343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.394156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.418214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.975331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.976350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.980591] Lustre: Skipped 2 previous similar messages [ 320.005600] Lustre: Skipped 2 previous similar messages [ 322.708630] Lustre: server umount lustre-MDT0000 complete [ 326.178052] LustreError: 7924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223095 with bad export cookie 8016896874171806472 [ 326.179321] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.191178] LustreError: 7924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 330.208397] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.210576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.211148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.211153] Lustre: Skipped 1 previous similar message [ 330.236034] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 330.309320] Lustre: Skipped 1 previous similar message [ 331.231742] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.259288] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 332.595348] Lustre: server umount lustre-MDT0001 complete [ 336.866538] Lustre: server umount lustre-OST0000 complete [ 340.659598] Lustre: server umount lustre-OST0001 complete [ 346.748401] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 355.020951] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 387.554854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.801073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 387.840533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 387.909257] Lustre: lustre-MDT0000: new disk, initializing [ 387.970032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.987514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 391.975119] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 401.987664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.222829] Lustre: 15069:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 402.359428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 402.365101] Lustre: Skipped 1 previous similar message [ 402.515311] Lustre: lustre-MDT0001: new disk, initializing [ 402.687059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 402.736309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 402.754115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.905536] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 417.434388] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 422.650594] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 428.985043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.317941] Lustre: lustre-OST0000: new disk, initializing [ 429.322407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.328850] Lustre: 17041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.418158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.974547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.990809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.082086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.840900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 444.302743] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 449.136923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.329644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.653195] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.865391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.673362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.684938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.693974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.703254] Lustre: Skipped 1 previous similar message [ 456.710860] Lustre: Skipped 3 previous similar messages [ 461.821870] Lustre: server umount lustre-OST0000 complete [ 466.916968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.931721] Lustre: Skipped 1 previous similar message [ 466.931752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.948382] Lustre: Skipped 2 previous similar messages [ 471.739477] Lustre: server umount lustre-MDT0000 complete [ 472.034149] LustreError: 15082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.049701] LustreError: 15082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 474.975194] LustreError: 17040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223243 with bad export cookie 8016896874171808005 [ 474.979451] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.989346] LustreError: 17040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 475.312827] Lustre: server umount lustre-MDT0001 complete [ 478.653301] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:00:45 (1782223245) [ 484.829483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.171148] LustreError: 19307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.241101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.646992] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 490.487509] LustreError: 19308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.891334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.543071] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 513.264236] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 518.980290] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 524.839163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 530.353162] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 536.248899] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 545.765885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.783694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 545.784481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.798720] LustreError: Skipped 1 previous similar message [ 549.152899] Lustre: server umount lustre-OST0000 complete [ 564.193545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.200433] Lustre: Skipped 6 previous similar messages [ 567.266217] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.421456] Lustre: server umount lustre-MDT0000 complete [ 570.342056] LustreError: 19302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.373640] LustreError: 19302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 571.278337] LustreError: 22098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223340 with bad export cookie 8016896874171808887 [ 571.289594] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.296406] LustreError: 22098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 571.764474] Lustre: server umount lustre-MDT0001 complete [ 582.986901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 585.730537] Key type lgssc unregistered [ 586.037739] LNet: 22851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.048033] LNetError: 22851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.072902] LNet: Removed LNI 192.168.204.126@tcp [ 587.121177] Key type .llcrypt unregistered [ 587.125797] Key type ._llcrypt unregistered [ 609.395330] Key type ._llcrypt registered [ 609.397405] Key type .llcrypt registered [ 609.475798] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 610.275245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 610.303969] alg: No test for adler32 (adler32-zlib) [ 611.363258] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 611.579569] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 613.215192] Key type lgssc registered [ 614.603850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.148239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.165163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.723124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.993237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 634.852547] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.876804] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 637.898018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.218060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 643.264188] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 651.228973] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 656.438993] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 664.136622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 664.477952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.631426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 670.394492] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 679.972953] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 699.250220] Lustre: server umount lustre-OST0000 complete [ 704.484529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 704.490083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 704.500651] Lustre: Skipped 1 previous similar message [ 704.514183] Lustre: Skipped 1 previous similar message [ 709.602112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.614189] Lustre: Skipped 1 previous similar message [ 713.914063] Lustre: server umount lustre-MDT0000 complete [ 714.721124] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.751197] LustreError: 24262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 715.744139] Lustre: 23574:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782223468/real 1782223468] req@0000000079566946 x1868796662935808/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782223484 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 715.788815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.813210] LustreError: 24263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 715.848597] LustreError: 24263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 717.611927] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223486 with bad export cookie 12034424056451791027 [ 717.614265] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.627693] LustreError: 24242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 717.981497] Lustre: server umount lustre-MDT0001 complete [ 728.412775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 731.483956] Key type lgssc unregistered [ 731.800291] LNet: 28239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.817866] LNetError: 28239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.841853] LNet: Removed LNI 192.168.204.126@tcp [ 732.783648] Key type .llcrypt unregistered [ 732.785523] Key type ._llcrypt unregistered [ 756.290758] Key type ._llcrypt registered [ 756.294354] Key type .llcrypt registered [ 756.386181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 757.642117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 757.656850] alg: No test for adler32 (adler32-zlib) [ 758.753893] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 759.109469] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 760.833288] Key type lgssc registered [ 762.189207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 771.011882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 771.036080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.686570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.063475] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 782.838921] LustreError: 29656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.858876] LustreError: 29656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 787.948774] LustreError: 29657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.819702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.066879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 795.700642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 803.812245] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 810.046786] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 824.578687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.972477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.858811] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 840.007897] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 842.434821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 852.974204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.975388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 853.004731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 853.020987] Lustre: Skipped 1 previous similar message [ 857.661691] Lustre: server umount lustre-OST0000 complete [ 862.714681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 862.728152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.743699] Lustre: Skipped 1 previous similar message [ 862.747267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.329721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.335641] Lustre: Skipped 3 previous similar messages [ 869.022224] Lustre: server umount lustre-MDT0000 complete [ 873.172192] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223642 with bad export cookie 4265804281490316738 [ 873.178881] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.182559] LustreError: 29636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 873.575244] Lustre: server umount lustre-MDT0001 complete [ 885.598191] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 889.248600] Key type lgssc unregistered [ 889.609881] LNet: 33535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.626059] LNetError: 33535:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.653300] LNet: Removed LNI 192.168.204.126@tcp [ 890.669173] Key type .llcrypt unregistered [ 890.674276] Key type ._llcrypt unregistered [ 916.133539] Key type ._llcrypt registered [ 916.137606] Key type .llcrypt registered [ 916.277660] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 917.300115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.403889] alg: No test for adler32 (adler32-zlib) [ 918.658475] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 918.959285] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 920.671179] Key type lgssc registered [ 921.805090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 930.447214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.477428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.044648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.864811] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 941.539241] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.566341] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 946.657741] LustreError: 34955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 946.860633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.367363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 951.541845] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 958.582851] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 964.489166] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 971.657251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 971.975525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.213716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 978.570604] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 985.696996] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 996.852974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.861538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 996.861752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 996.876826] Lustre: Skipped 1 previous similar message [ 1001.402663] Lustre: server umount lustre-OST0000 complete [ 1007.073411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.079155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.094516] Lustre: Skipped 1 previous similar message [ 1007.110236] Lustre: Skipped 2 previous similar messages [ 1011.531177] Lustre: server umount lustre-MDT0000 complete [ 1012.194375] LustreError: 34955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.215596] LustreError: 34955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1015.405684] LustreError: 34935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223784 with bad export cookie 12873968047892323943 [ 1015.413851] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.421705] LustreError: 34935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1015.730796] Lustre: server umount lustre-MDT0001 complete [ 1027.740120] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1030.966650] Key type lgssc unregistered [ 1031.458536] LNet: 38491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.468492] LNetError: 38491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.553706] LNet: Removed LNI 192.168.204.126@tcp [ 1033.470873] Key type .llcrypt unregistered [ 1033.481272] Key type ._llcrypt unregistered [ 1056.715687] Key type ._llcrypt registered [ 1056.718924] Key type .llcrypt registered [ 1056.807034] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1058.178184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.202196] alg: No test for adler32 (adler32-zlib) [ 1059.325110] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 1059.494503] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1061.159206] Key type lgssc registered [ 1062.415105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.771654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1070.807305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.191875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.088369] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1082.341149] LustreError: 39904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.363650] LustreError: 39904:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1086.232026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.653424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1091.704489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1099.466954] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.269309] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.459595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.890330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.010787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1122.820503] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1131.088390] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1138.665085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1138.675712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.696580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.728896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.731377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.736207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.771431] Lustre: Skipped 2 previous similar messages [ 1144.210193] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1144.571085] Lustre: server umount lustre-OST0000 complete [ 1168.867858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1168.881315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.905897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1171.442665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.114245] Lustre: server umount lustre-MDT0000 complete [ 1176.560747] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1176.600741] LustreError: 39900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1178.044712] LustreError: 39884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782223947 with bad export cookie 6612593425661168865 [ 1178.065569] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1184.607780] Lustre: server umount lustre-MDT0001 complete [ 1199.732378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1202.493583] Key type lgssc unregistered [ 1202.904180] LNet: 43615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.914458] LNetError: 43615:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.931802] LNet: Removed LNI 192.168.204.126@tcp [ 1204.010774] Key type .llcrypt unregistered [ 1204.015623] Key type ._llcrypt unregistered [ 1229.318776] Key type ._llcrypt registered [ 1229.325306] Key type .llcrypt registered [ 1229.506095] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1230.954402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1230.987340] alg: No test for adler32 (adler32-zlib) [ 1232.208974] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 1232.440240] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1234.151533] Key type lgssc registered [ 1235.659435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1245.123689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1245.148865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.742678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.906840] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1256.930019] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.950291] LustreError: 45030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1257.953056] LustreError: 45029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.932172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.408163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.138768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1275.892600] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.485606] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1290.517351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.793422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.850473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1296.734743] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1304.835111] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1312.227728] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.229573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1312.253835] Lustre: Skipped 1 previous similar message [ 1317.351054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1317.364987] Lustre: Skipped 5 previous similar messages [ 1318.508547] Lustre: server umount lustre-MDT0000 complete [ 1322.470248] LustreError: 45757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1322.504734] LustreError: 45757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1322.596151] LustreError: 45011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224091 with bad export cookie 12177640961773368126 [ 1322.601824] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1322.605287] LustreError: 45011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1322.930575] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1322.939086] Lustre: Skipped 1 previous similar message [ 1327.588844] LustreError: 46516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.591103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.618919] LustreError: 46516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1327.655657] Lustre: Skipped 1 previous similar message [ 1327.675516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1329.048919] Lustre: server umount lustre-MDT0001 complete [ 1350.827811] Lustre: server umount lustre-OST0000 complete [ 1368.894076] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1372.056511] Key type lgssc unregistered [ 1372.459650] LNet: 48893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.477391] LNetError: 48893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.489776] LNet: Removed LNI 192.168.204.126@tcp [ 1373.535749] Key type .llcrypt unregistered [ 1373.538928] Key type ._llcrypt unregistered [ 1403.053915] Key type ._llcrypt registered [ 1403.056165] Key type .llcrypt registered [ 1403.209943] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1404.784663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1404.843445] alg: No test for adler32 (adler32-zlib) [ 1406.080966] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 1406.598575] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1408.335280] Key type lgssc registered [ 1409.742290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.023529] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1419.036304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.591392] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1439.711338] LustreError: 50279:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 1466.399198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.748699] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1482.407564] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1649.120717] Lustre: server umount lustre-OST0000 complete [ 1669.365889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1672.607570] Key type lgssc unregistered [ 1672.934892] LNet: 51962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.949644] LNetError: 51962:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.969850] LNet: Removed LNI 192.168.204.126@tcp [ 1673.806513] Key type .llcrypt unregistered [ 1673.809954] Key type ._llcrypt unregistered [ 1701.735976] Key type ._llcrypt registered [ 1701.742390] Key type .llcrypt registered [ 1701.886267] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1702.888279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.006624] alg: No test for adler32 (adler32-zlib) [ 1704.140194] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 1704.459944] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1706.232753] Key type lgssc registered [ 1707.748144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1716.531453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1716.571608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.047820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.068051] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1727.457116] LustreError: 53378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1727.472581] LustreError: 53378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1732.578323] LustreError: 53379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.094893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.500848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1737.680481] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1746.715835] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1753.293048] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1761.759109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.024676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.066560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1768.865848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1777.043506] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.777367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.804450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.813638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1785.911433] Lustre: server umount lustre-OST0000 complete [ 1789.920664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1789.935587] LustreError: Skipped 1 previous similar message [ 1789.954418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.976191] Lustre: Skipped 1 previous similar message [ 1789.982325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.994240] Lustre: Skipped 1 previous similar message [ 1794.016621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.017676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.930433] Lustre: server umount lustre-MDT0000 complete [ 1799.140744] LustreError: 54106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1799.175015] LustreError: 54106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1800.334157] LustreError: 53358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224569 with bad export cookie 7180612537185074290 [ 1800.341673] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1800.362240] LustreError: 53358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1800.658321] Lustre: server umount lustre-MDT0001 complete [ 1814.005735] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1818.167435] Key type lgssc unregistered [ 1818.531442] LNet: 56913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.542417] LNetError: 56913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.562871] LNet: Removed LNI 192.168.204.126@tcp [ 1819.728502] Key type .llcrypt unregistered [ 1819.730264] Key type ._llcrypt unregistered [ 1845.339573] Key type ._llcrypt registered [ 1845.346838] Key type .llcrypt registered [ 1845.570550] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1846.734385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.762281] alg: No test for adler32 (adler32-zlib) [ 1848.307178] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 1848.645709] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1850.359212] Key type lgssc registered [ 1851.578630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1861.173505] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1861.226223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1876.895443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1882.015312] LustreError: 58297:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 1908.703313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.777655] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1924.206074] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1932.389157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.969858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.066881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1937.366921] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1938.408253] LustreError: 58322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.432652] LustreError: 58322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1943.521416] LustreError: 58323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.552369] LustreError: 58323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1947.037186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.397281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.138398] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1961.780554] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1969.091581] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1973.217314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1973.234078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.260375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1977.629380] Lustre: server umount lustre-OST0000 complete [ 2003.940696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.943638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.955656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.976336] Lustre: Skipped 2 previous similar messages [ 2003.991364] Lustre: Skipped 2 previous similar messages [ 2005.525345] Lustre: server umount lustre-MDT0000 complete [ 2009.057380] LustreError: 59338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.871210] LustreError: 58305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224778 with bad export cookie 8371201110820537051 [ 2009.879799] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.889338] LustreError: 58305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2016.383789] Lustre: server umount lustre-MDT0001 complete [ 2027.708748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2030.606321] Key type lgssc unregistered [ 2030.878621] LNet: 62040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.886517] LNetError: 62040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.903705] LNet: Removed LNI 192.168.204.126@tcp [ 2031.798850] Key type .llcrypt unregistered [ 2031.801279] Key type ._llcrypt unregistered [ 2058.776600] Key type ._llcrypt registered [ 2058.779663] Key type .llcrypt registered [ 2058.923839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2059.684339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2059.804122] alg: No test for adler32 (adler32-zlib) [ 2061.219982] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2061.656996] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2063.351513] Key type lgssc registered [ 2065.830801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2075.502909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2075.534219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.102210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.030376] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2082.277729] LustreError: 63453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.316321] LustreError: 63453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2087.401278] LustreError: 63454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.080165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.616293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2097.167852] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2106.293365] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2113.627893] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2120.523417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2120.957244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.008288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2127.863827] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2136.648436] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2158.048738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2158.053255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2158.073121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2158.094114] Lustre: Skipped 1 previous similar message [ 2158.119869] Lustre: Skipped 1 previous similar message [ 2163.170818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.187058] Lustre: Skipped 1 previous similar message [ 2168.303155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.311180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2169.541599] Lustre: server umount lustre-OST0000 complete [ 2178.534179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2178.536525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.564713] Lustre: Skipped 1 previous similar message [ 2178.575722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.584992] Lustre: Skipped 2 previous similar messages [ 2179.882954] Lustre: server umount lustre-MDT0000 complete [ 2183.593525] LustreError: 64191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782224952 with bad export cookie 16951878616940880049 [ 2183.596408] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2183.600839] LustreError: 64191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2184.004640] Lustre: server umount lustre-MDT0001 complete [ 2195.203326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2197.943191] Key type lgssc unregistered [ 2198.240685] LNet: 67015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.252593] LNetError: 67015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.296337] LNet: Removed LNI 192.168.204.126@tcp [ 2199.107966] Key type .llcrypt unregistered [ 2199.111098] Key type ._llcrypt unregistered [ 2227.469951] Key type ._llcrypt registered [ 2227.473392] Key type .llcrypt registered [ 2227.630908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2228.872184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2228.909331] alg: No test for adler32 (adler32-zlib) [ 2229.991889] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2230.347941] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2232.079190] Key type lgssc registered [ 2233.130828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2241.539181] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:30:08 (1782225008) [ 2248.997463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2249.028531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.671176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2255.678941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2260.971171] LustreError: 68721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.995547] LustreError: 68721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2265.289083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.687647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.828323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2279.084121] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2286.025569] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2293.358568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.843299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2295.923571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2300.287769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2308.243412] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2317.275726] Lustre: Failing over lustre-MDT0000 [ 2317.619796] Lustre: server umount lustre-MDT0000 complete [ 2320.303777] LustreError: 71213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.877114] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.900235] Lustre: Skipped 2 previous similar messages [ 2325.386821] LustreError: 69690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.414431] LustreError: 69690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2327.460960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.630824] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.754943] Lustre: *** cfs_fail_loc=135, val=0*** [ 2327.859487] LustreError: 71809:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2327.871860] LustreError: 71809:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.126@tcp: cfg command failed: rc = -2 [ 2327.884172] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2327.884172] [ 2327.893833] LustreError: MGC192.168.204.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2327.908735] LustreError: 71798:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2327.916583] LustreError: 71798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2327.936026] LustreError: 71798:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2327.946577] LustreError: 71798:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2327.977110] Lustre: server umount lustre-MDT0000 complete [ 2327.979975] LustreError: 71798:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2330.512564] LustreError: 68716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.536581] LustreError: 68716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2335.634858] LustreError: 68716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.656875] LustreError: 68716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2335.713091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2335.918259] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2335.934432] LustreError: 68714:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2336.245662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2336.262350] Lustre: Skipped 2 previous similar messages [ 2336.488101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.574206] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2336.715845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2341.294983] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2341.884983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2341.949508] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2342.030856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2351.584838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2351.595632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.624579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.225534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.236948] Lustre: Skipped 3 previous similar messages [ 2357.563318] Lustre: server umount lustre-OST0000 complete [ 2362.335879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2362.336745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.340155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.372055] Lustre: Skipped 2 previous similar messages [ 2367.456635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.469688] Lustre: Skipped 2 previous similar messages [ 2367.729497] Lustre: server umount lustre-MDT0000 complete [ 2371.918814] LustreError: 68703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225140 with bad export cookie 12038004098823301351 [ 2371.922304] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2371.928546] LustreError: 68703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2372.179705] Lustre: server umount lustre-MDT0001 complete [ 2384.077552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2388.276678] Key type lgssc unregistered [ 2388.599813] LNet: 73587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.608700] LNetError: 73587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.630428] LNet: Removed LNI 192.168.204.126@tcp [ 2389.914167] Key type .llcrypt unregistered [ 2389.918928] Key type ._llcrypt unregistered [ 2417.919908] Key type ._llcrypt registered [ 2417.922131] Key type .llcrypt registered [ 2418.115874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2419.671645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.024498] alg: No test for adler32 (adler32-zlib) [ 2421.478789] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2421.939923] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2423.727112] Key type lgssc registered [ 2424.841744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.155488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.211860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.658199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.393980] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2446.827291] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.839117] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2447.844288] LustreError: 75002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.091559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.486608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2455.248576] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2463.195876] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.908927] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2475.664332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.943532] LustreError: 76822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.095652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.162887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2482.673844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2490.521759] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.679421] Lustre: Failing over lustre-MDT0000 [ 2495.885624] Lustre: server umount lustre-MDT0000 complete [ 2496.485900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.498455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.514710] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.511401] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.522395] Lustre: Skipped 1 previous similar message [ 2502.538646] LustreError: 76435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.550666] LustreError: 76435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2504.820805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.927738] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.062900] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2505.245937] LustreError: 78090:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2505.253130] LustreError: 78090:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.126@tcp: cfg command failed: rc = -9 [ 2505.264249] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2505.264249] [ 2505.278324] LustreError: MGC192.168.204.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2505.297352] LustreError: 78078:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2505.308635] LustreError: 78078:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2505.322522] LustreError: 78078:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2505.326562] LustreError: 78078:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2505.355458] Lustre: server umount lustre-MDT0000 complete [ 2505.362769] LustreError: 78078:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2512.425801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.583965] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2512.909369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.968977] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2517.786521] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2517.922653] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2517.995120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2518.043246] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2518.149798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2528.233972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.235822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.245176] Lustre: Skipped 1 previous similar message [ 2533.362495] LustreError: 76825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.404267] LustreError: 76825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2533.549148] Lustre: server umount lustre-OST0000 complete [ 2538.467319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2538.469274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.470518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.470523] Lustre: Skipped 1 previous similar message [ 2538.485509] Lustre: Skipped 1 previous similar message [ 2543.585880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.594433] Lustre: Skipped 2 previous similar messages [ 2548.719724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.728640] Lustre: Skipped 1 previous similar message [ 2551.263828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2551.504520] Lustre: server umount lustre-MDT0000 complete [ 2553.827118] LustreError: 74996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.840022] LustreError: 74996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2555.320236] LustreError: 74981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225324 with bad export cookie 4540338563870810288 [ 2555.334140] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.336567] LustreError: 74981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2555.712605] Lustre: server umount lustre-MDT0001 complete [ 2566.689675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2569.563605] Key type lgssc unregistered [ 2569.951511] LNet: 79871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.956683] LNetError: 79871:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.974920] LNet: Removed LNI 192.168.204.126@tcp [ 2571.144274] Key type .llcrypt unregistered [ 2571.146709] Key type ._llcrypt unregistered [ 2584.053760] Key type ._llcrypt registered [ 2584.057922] Key type .llcrypt registered [ 2584.198438] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2584.548492] Key type .llcrypt unregistered [ 2584.550207] Key type ._llcrypt unregistered [ 2613.982268] Key type ._llcrypt registered [ 2613.984673] Key type .llcrypt registered [ 2614.115596] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2615.210613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.223332] alg: No test for adler32 (adler32-zlib) [ 2616.315460] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2616.553550] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2618.247469] Key type lgssc registered [ 2619.484995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2627.840917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.860966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.451617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.027647] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2639.843111] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.862378] LustreError: 81775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2643.350584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.766307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.571518] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2656.588190] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.253845] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2669.035993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2669.420702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.546735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2676.196719] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2684.038443] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2699.233128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.237177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.252814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.265494] Lustre: Skipped 1 previous similar message [ 2699.287980] Lustre: Skipped 1 previous similar message [ 2700.303306] Lustre: server umount lustre-OST0000 complete [ 2705.376363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2705.392535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.409395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.475985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.477244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2710.737482] Lustre: server umount lustre-MDT0000 complete [ 2714.377982] LustreError: 81755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225483 with bad export cookie 14420635160082864337 [ 2714.379343] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.387486] LustreError: 81755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2714.849657] Lustre: server umount lustre-MDT0001 complete [ 2725.903572] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2728.744756] Key type lgssc unregistered [ 2729.080972] LNet: 85333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.092531] LNetError: 85333:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.111971] LNet: Removed LNI 192.168.204.126@tcp [ 2730.067982] Key type .llcrypt unregistered [ 2730.071055] Key type ._llcrypt unregistered [ 2753.697848] Key type ._llcrypt registered [ 2753.699699] Key type .llcrypt registered [ 2753.800302] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2754.823224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2755.128093] alg: No test for adler32 (adler32-zlib) [ 2756.207629] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2756.424196] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2758.103422] Key type lgssc registered [ 2759.299574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.081620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.101035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.708124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.686713] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2778.091933] LustreError: 86744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.124166] LustreError: 86744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2782.140827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.439869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.842239] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2795.975126] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.419786] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2809.446257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2809.788587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2813.892863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2815.972517] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2823.922413] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2837.472443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2837.474781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.487560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.499609] Lustre: Skipped 1 previous similar message [ 2837.519975] Lustre: Skipped 1 previous similar message [ 2840.677762] Lustre: server umount lustre-OST0000 complete [ 2847.716990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2847.723934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.728854] Lustre: Skipped 1 previous similar message [ 2847.742665] Lustre: Skipped 1 previous similar message [ 2850.905464] Lustre: server umount lustre-MDT0000 complete [ 2852.832389] LustreError: 87473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.854453] LustreError: 87473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2854.824167] LustreError: 86724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225623 with bad export cookie 4543864230051819174 [ 2854.846917] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2855.204042] Lustre: server umount lustre-MDT0001 complete [ 2866.059446] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2868.703729] Key type lgssc unregistered [ 2869.010353] LNet: 90278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2869.027267] LNetError: 90278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2869.046728] LNet: Removed LNI 192.168.204.126@tcp [ 2869.983934] Key type .llcrypt unregistered [ 2869.986988] Key type ._llcrypt unregistered [ 2895.496857] Key type ._llcrypt registered [ 2895.501827] Key type .llcrypt registered [ 2895.628416] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2896.505709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2896.626247] alg: No test for adler32 (adler32-zlib) [ 2897.727696] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 2898.062865] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2899.759826] Key type lgssc registered [ 2901.062023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2909.263480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2909.287390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.696410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.104598] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2920.934472] LustreError: 91694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.963778] LustreError: 91694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2923.288397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.698317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2929.479491] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2938.154265] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2945.459733] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2955.611921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.755062] LustreError: 93512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.160996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2961.420843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2963.686968] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2971.722935] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2985.962890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2985.972378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.987423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.034348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2989.045055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.060591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2989.077190] Lustre: Skipped 1 previous similar message [ 2990.713792] Lustre: server umount lustre-OST0000 complete [ 2995.168937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2995.177191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.209411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.277450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.279516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.294883] Lustre: Skipped 1 previous similar message [ 3001.173127] Lustre: server umount lustre-MDT0000 complete [ 3004.386534] LustreError: 91688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.400121] LustreError: 91688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3005.046619] LustreError: 91674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782225774 with bad export cookie 11441674050742601398 [ 3005.052396] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.058825] LustreError: 91674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3005.516889] Lustre: server umount lustre-MDT0001 complete [ 3018.337514] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3021.258267] Key type lgssc unregistered [ 3021.540385] LNet: 95256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3021.546943] LNetError: 95256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3022.569303] LNet: Removed LNI 192.168.204.126@tcp [ 3023.462602] Key type .llcrypt unregistered [ 3023.466146] Key type ._llcrypt unregistered [ 3047.075540] Key type ._llcrypt registered [ 3047.077600] Key type .llcrypt registered [ 3047.217282] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3048.358230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3048.442427] alg: No test for adler32 (adler32-zlib) [ 3049.533500] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 3049.835110] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3051.503169] Key type lgssc registered [ 3052.798569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3060.914349] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3060.939867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.511616] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3081.632728] LustreError: 96641:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 3108.384448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3114.659233] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3122.825265] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.869891] Lustre: server umount lustre-OST0000 complete [ 3138.334652] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:45:04 (1782225904) [ 3144.855347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.325899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.043426] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3150.821586] LustreError: 98279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.851141] LustreError: 98279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3155.943689] LustreError: 98298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.312991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.752551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.868227] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3172.451442] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.518577] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3185.605858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.941345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.066180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3191.254449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3198.522507] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3214.821354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.826020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3214.843479] Lustre: Skipped 1 previous similar message [ 3214.853617] Lustre: Skipped 1 previous similar message [ 3219.642929] Lustre: server umount lustre-OST0000 complete [ 3225.063547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3225.067677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3225.079360] Lustre: Skipped 1 previous similar message [ 3229.663097] Lustre: server umount lustre-MDT0000 complete [ 3230.176609] LustreError: 98274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.188316] LustreError: 98274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3233.213537] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226002 with bad export cookie 10604834572320818811 [ 3233.223541] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3233.226600] LustreError: 98261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3233.542093] Lustre: server umount lustre-MDT0001 complete [ 3244.737133] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3247.471757] Key type lgssc unregistered [ 3247.756805] LNet: 101971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3247.763880] LNetError: 101971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3248.808153] LNet: Removed LNI 192.168.204.126@tcp [ 3249.678933] Key type .llcrypt unregistered [ 3249.682531] Key type ._llcrypt unregistered [ 3291.159306] Key type ._llcrypt registered [ 3291.164065] Key type .llcrypt registered [ 3291.332278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3293.049742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3293.095982] alg: No test for adler32 (adler32-zlib) [ 3294.410367] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 3294.898452] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3296.673491] Key type lgssc registered [ 3298.192230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.180743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3307.215474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.778264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3313.025662] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3318.249839] LustreError: 104160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.269561] LustreError: 104160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3322.629874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.934308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3327.319357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3335.919397] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3342.691860] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3349.955915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3350.332734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.401731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3357.554665] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3366.133983] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3378.155322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.156449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3378.170901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.191412] Lustre: Skipped 1 previous similar message [ 3380.763921] Lustre: server umount lustre-OST0000 complete [ 3384.291534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3384.300318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3384.311897] Lustre: Skipped 1 previous similar message [ 3384.322177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.386495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3388.388409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.356861] Lustre: server umount lustre-MDT0000 complete [ 3393.512964] LustreError: 104161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.535882] LustreError: 104161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3393.862772] LustreError: 104140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226162 with bad export cookie 14958790467873567276 [ 3393.863076] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.878221] LustreError: 104140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3394.180238] Lustre: server umount lustre-MDT0001 complete [ 3404.512190] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3406.952614] Key type lgssc unregistered [ 3407.312196] LNet: 107696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3407.322609] LNetError: 107696:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3407.336992] LNet: Removed LNI 192.168.204.126@tcp [ 3408.040914] Key type .llcrypt unregistered [ 3408.050652] Key type ._llcrypt unregistered [ 3427.137556] Key type ._llcrypt registered [ 3427.139685] Key type .llcrypt registered [ 3427.229334] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3428.503223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3428.554010] alg: No test for adler32 (adler32-zlib) [ 3429.627205] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 3429.893996] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3431.583251] Key type lgssc registered [ 3432.650848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3440.371546] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3440.399982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3455.968260] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3461.087507] LustreError: 108945:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 3487.775275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3495.264955] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3503.070639] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3510.978913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.437555] LustreError: MGC192.168.204.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3511.447930] LustreError: 109969:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3511.452625] LustreError: 109969:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3511.458875] LustreError: 109969:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3511.639471] Lustre: server umount lustre-MDT0000 complete [ 3511.641412] LustreError: 109969:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3531.744528] Lustre: 108282:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782226284/real 1782226284] req@0000000014722507 x1868799617797504/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782226300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3531.769227] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.781554] LustreError: 108957:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3552.297531] Lustre: server umount lustre-OST0000 complete [ 3561.613420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3572.393290] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3612.488779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.832168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3612.875892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3612.976253] Lustre: lustre-MDT0000: new disk, initializing [ 3613.044489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3613.070748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3617.727329] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3628.674846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.790866] Lustre: 112767:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3628.850120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3628.854717] Lustre: Skipped 1 previous similar message [ 3628.864579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3628.929438] Lustre: lustre-MDT0001: new disk, initializing [ 3628.994276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3629.040721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3629.055639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3634.370814] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3644.489507] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3650.321835] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3657.451679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.699729] Lustre: lustre-OST0000: new disk, initializing [ 3657.703872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3657.708509] Lustre: 114733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3657.777738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.187591] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3664.480455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3664.492935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3664.651779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3672.398498] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3677.713154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3677.909779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3683.079545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3683.367816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3690.463909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3690.470031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.474072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3690.478155] Lustre: Skipped 1 previous similar message [ 3690.491623] Lustre: Skipped 1 previous similar message [ 3691.822711] Lustre: server umount lustre-OST0000 complete [ 3700.707713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.719266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3700.724155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.146686] Lustre: server umount lustre-MDT0000 complete [ 3706.335314] LustreError: 112757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226475 with bad export cookie 11095836354616521408 [ 3706.339288] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3706.358301] LustreError: 112757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3706.898590] Lustre: server umount lustre-MDT0001 complete [ 3715.915613] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:54:42 (1782226482) [ 3719.298868] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3736.123870] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3745.136634] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3780.764586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.994137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3781.038419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3781.131616] Lustre: lustre-MDT0000: new disk, initializing [ 3781.204512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.243411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3785.772328] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3798.372560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3798.535892] Lustre: 119635:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3798.628158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3798.634615] Lustre: Skipped 1 previous similar message [ 3798.754800] Lustre: lustre-MDT0001: new disk, initializing [ 3798.877901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3798.909482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3798.925564] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3805.055228] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3815.211848] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3822.310488] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3828.987963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.447639] Lustre: lustre-OST0000: new disk, initializing [ 3829.463804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3829.482443] Lustre: 121602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3829.615819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.621770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3838.030032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3838.040629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3838.164815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3846.283892] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3852.352883] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3852.539540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3857.815269] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3858.030736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3864.035076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3864.045861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3864.052436] LustreError: Skipped 1 previous similar message [ 3864.053868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3864.067962] Lustre: Skipped 2 previous similar messages [ 3864.078465] Lustre: Skipped 2 previous similar messages [ 3869.156262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.169841] Lustre: Skipped 1 previous similar message [ 3874.274271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3874.467252] Lustre: server umount lustre-OST0000 complete [ 3879.393904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.396410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.402876] Lustre: Skipped 1 previous similar message [ 3879.409455] Lustre: Skipped 3 previous similar messages [ 3885.344641] Lustre: server umount lustre-MDT0000 complete [ 3889.322924] LustreError: 119628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226658 with bad export cookie 11095836354616522290 [ 3889.334139] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3889.336646] LustreError: 119628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3889.840369] Lustre: server umount lustre-MDT0001 complete [ 3899.324667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.747986] LustreError: 123822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3899.761868] LustreError: 123822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3899.816522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.692508] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3904.996442] LustreError: 123823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.090569] LustreError: 123822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3914.039412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.415738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3927.836460] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3934.155219] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3942.265862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3949.867275] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3958.313633] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3966.562654] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3973.550954] Lustre: server umount lustre-OST0000 complete [ 3980.769461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3980.773140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.790034] Lustre: Skipped 1 previous similar message [ 3980.799403] Lustre: Skipped 2 previous similar messages [ 3980.811218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.042483] Lustre: server umount lustre-MDT0000 complete [ 3991.017030] LustreError: 124800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.043096] LustreError: 124800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3991.051565] Lustre: 108282:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782226744/real 1782226744] req@0000000012065de3 x1868799617912704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782226760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3991.077690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.852249] LustreError: 123804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226760 with bad export cookie 11095836354616523172 [ 3991.867229] LustreError: 123804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3991.869365] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3992.219298] Lustre: server umount lustre-MDT0001 complete [ 4002.985342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4005.422107] Key type lgssc unregistered [ 4005.690910] LNet: 127451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4005.699742] LNetError: 127451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4005.711701] LNet: Removed LNI 192.168.204.126@tcp [ 4006.514683] Key type .llcrypt unregistered [ 4006.520663] Key type ._llcrypt unregistered [ 4023.973883] Key type ._llcrypt registered [ 4023.976325] Key type .llcrypt registered [ 4024.063086] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4039.287741] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4040.870242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4040.930510] alg: No test for adler32 (adler32-zlib) [ 4041.951421] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 4042.199120] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4043.911186] Key type lgssc registered [ 4044.947450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.402499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.418141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.725612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4086.758941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4086.867362] Lustre: lustre-MDT0000: new disk, initializing [ 4087.001115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.024441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4091.557440] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4102.057194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4102.222964] Lustre: 130608:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4102.298774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4102.306866] Lustre: Skipped 1 previous similar message [ 4102.369557] Lustre: lustre-MDT0001: new disk, initializing [ 4102.465446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4102.510040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4102.525255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4107.232681] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4118.707039] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4125.285835] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4132.678052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4132.960424] Lustre: lustre-OST0000: new disk, initializing [ 4132.964743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4132.969625] Lustre: 132589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4133.078688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.440786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4141.158359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4141.169842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4141.263863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4150.162962] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4155.484633] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4155.791048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4160.284632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4160.537519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4167.148982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.154778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4167.176973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.187450] Lustre: Skipped 1 previous similar message [ 4172.265100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.273391] Lustre: Skipped 2 previous similar messages [ 4177.375462] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4177.376744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4177.398886] Lustre: Skipped 1 previous similar message [ 4177.604959] Lustre: server umount lustre-OST0000 complete [ 4182.496677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.500086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.528501] Lustre: Skipped 1 previous similar message [ 4182.545228] Lustre: Skipped 1 previous similar message [ 4187.616698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.628074] Lustre: Skipped 1 previous similar message [ 4196.319141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4196.643727] Lustre: server umount lustre-MDT0000 complete [ 4200.823264] LustreError: 130601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782226969 with bad export cookie 13439392534528699477 [ 4200.828096] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4200.838193] LustreError: 130601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4201.167456] Lustre: server umount lustre-MDT0001 complete [ 4211.514365] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:02:57 (1782226977) [ 4218.533234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.051181] LustreError: 135004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.067952] LustreError: 135004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4219.124906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.079928] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4224.498112] LustreError: 135005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.604537] LustreError: 135004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.307036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.819630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4239.503502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4247.899175] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4254.455433] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4259.683960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.706179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.718501] Lustre: Skipped 3 previous similar messages [ 4262.880364] Lustre: server umount lustre-MDT0000 complete [ 4264.938957] LustreError: 135005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.965467] LustreError: 135005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4266.506603] LustreError: 134985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227035 with bad export cookie 13439392534528700352 [ 4266.520830] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.522884] LustreError: 134985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.770650] Lustre: server umount lustre-MDT0001 complete [ 4276.036754] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:04:02 (1782227042) [ 4282.788384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.335594] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4303.457123] LustreError: 137420:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 4329.183941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4335.974265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4345.048647] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4347.575209] Lustre: server umount lustre-OST0000 complete [ 4357.726763] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:05:23 (1782227123) [ 4364.515701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.907479] LustreError: 138867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.969883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.433782] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4374.497558] LustreError: 138868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.513374] LustreError: 138868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4378.943226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4384.451625] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4392.713634] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4398.652909] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4405.644180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.026337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.034020] Lustre: Skipped 1 previous similar message [ 4412.907514] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4421.367253] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4435.952412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4435.962380] LustreError: Skipped 1 previous similar message [ 4435.966559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.979149] Lustre: Skipped 1 previous similar message [ 4435.990379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.035974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.054723] Lustre: Skipped 1 previous similar message [ 4441.429669] Lustre: server umount lustre-MDT0000 complete [ 4445.159068] LustreError: 139864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4445.187871] LustreError: 139864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4445.853115] LustreError: 138847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227214 with bad export cookie 13439392534528701017 [ 4445.862447] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4445.869062] LustreError: 138847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4446.235533] Lustre: server umount lustre-MDT0001 complete [ 4460.969873] Lustre: server umount lustre-OST0000 complete [ 4469.573626] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:07:16 (1782227236) [ 4476.689775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4477.193712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.343337] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4482.538924] LustreError: 142545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.569757] LustreError: 142545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4491.611543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4496.709854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4504.764960] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4511.426260] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4518.105438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.564057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4525.178608] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4533.035278] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4538.358205] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4538.602735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4543.597162] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4543.788411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4548.068097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.082589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.083279] LustreError: Skipped 1 previous similar message [ 4548.084768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.084774] Lustre: Skipped 3 previous similar messages [ 4548.119430] Lustre: Skipped 1 previous similar message [ 4560.351323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4560.598977] Lustre: server umount lustre-OST0000 complete [ 4568.545904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.563170] Lustre: Skipped 1 previous similar message [ 4573.537841] Lustre: server umount lustre-MDT0000 complete [ 4573.668184] LustreError: 142544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.690037] LustreError: 142544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4577.136467] LustreError: 142525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227346 with bad export cookie 13439392534528702095 [ 4577.141253] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4577.148973] LustreError: 142525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4577.491675] Lustre: server umount lustre-MDT0001 complete [ 4586.530647] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:09:12 (1782227352) [ 4593.498294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.055584] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4614.177536] LustreError: 146704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 4639.903429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.908675] Lustre: Skipped 2 previous similar messages [ 4645.786310] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4654.344877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4662.035247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.677249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4667.855883] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4677.702537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.049425] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4691.900164] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4697.823989] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4702.613331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4702.793314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4707.560328] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4707.776408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4713.962069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.968110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4713.980907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4713.995984] Lustre: Skipped 9 previous similar messages [ 4724.194367] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4724.472396] Lustre: server umount lustre-MDT0000 complete [ 4729.319416] LustreError: 149077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.344621] LustreError: 149077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4729.374098] LustreError: 146711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227498 with bad export cookie 13439392534528703117 [ 4729.381201] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4729.383559] LustreError: 146711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4729.645571] Lustre: server umount lustre-MDT0001 complete [ 4743.895396] Lustre: server umount lustre-OST0000 complete [ 4752.420670] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:11:58 (1782227518) [ 4758.679528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4774.304363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4779.425946] LustreError: 150902:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 4811.497850] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4819.690026] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4826.377177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.855210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4831.190738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4839.874370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.186933] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4853.522300] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4859.402313] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4866.132263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.271245] Lustre: lustre-OST0001: new disk, initializing [ 4866.274825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4866.280271] Lustre: 153773:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4872.546488] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4873.777951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4873.791601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4873.905698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4882.070837] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4887.537127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4887.736817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4892.448474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4892.745915] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4896.736665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4896.760307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.783776] Lustre: Skipped 2 previous similar messages [ 4896.793767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4896.799838] Lustre: Skipped 6 previous similar messages [ 4901.083512] Lustre: server umount lustre-OST0000 complete [ 4919.265343] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4919.429572] Lustre: server umount lustre-OST0001 complete [ 4929.627367] Lustre: server umount lustre-MDT0000 complete [ 4933.598728] LustreError: 150910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227702 with bad export cookie 13439392534528704069 [ 4933.607245] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.616552] LustreError: 150910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4933.994422] Lustre: server umount lustre-MDT0001 complete [ 4958.227308] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:15:24 (1782227724) [ 4959.558404] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4961.113879] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4962.672370] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:15:29 (1782227729) [ 4969.022347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.248322] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4969.270429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4969.409057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.415287] Lustre: Skipped 6 previous similar messages [ 4973.758644] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4981.957466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4982.091803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4982.118453] Lustre: 157333:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4986.649758] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4994.363794] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5000.754294] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5007.625346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5007.798477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5014.200666] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5014.666302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5022.053114] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5027.392969] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5027.611210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5032.261754] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5032.521150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5034.987320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5034.999040] LustreError: Skipped 1 previous similar message [ 5035.003163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.030930] Lustre: Skipped 5 previous similar messages [ 5040.629182] Lustre: server umount lustre-OST0000 complete [ 5063.544910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.177358] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5075.181549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.219910] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5084.277072] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.507458] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.543435] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5094.611692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5095.021969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5100.004217] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5100.303029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5107.652992] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d83c6437800.ost_server_uuid 50 [ 5109.691663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8d83c6437800.ost_server_uuid in IDLE state after 0 sec [ 5110.097410] LustreError: 157338:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.26@tcp) returned error from blocking AST (req@00000000f19ee7a7 x1868800260846848 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000161f2afc/0xba82504e219ec79f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.26@tcp remote: 0xe6c187b0b507da expref: 5 pid: 157340 timeout: 5210 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5110.154218] LustreError: lustre-MDT0000: A client on nid 192.168.204.26@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5110.166131] LustreError: 157328:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.26@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000161f2afc/0xba82504e219ec79f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.26@tcp remote: 0xe6c187b0b507da expref: 6 pid: 157340 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5115.883165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5115.909395] LustreError: Skipped 1 previous similar message [ 5121.515565] LustreError: 160964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5121.542638] LustreError: 160964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 5121.802848] Lustre: server umount lustre-OST0000 complete [ 5132.133176] Lustre: server umount lustre-MDT0000 complete [ 5136.722707] LustreError: 158074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782227905 with bad export cookie 13439392534528705252 [ 5136.725755] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.735036] LustreError: 158074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5151.199894] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5151.593053] Lustre: server umount lustre-MDT0001 complete [ 5162.960990] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5165.696639] Key type lgssc unregistered [ 5165.865869] LNet: 163452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5165.871196] LNetError: 163452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5165.883708] LNet: Removed LNI 192.168.204.126@tcp [ 5166.577438] Key type .llcrypt unregistered [ 5166.579678] Key type ._llcrypt unregistered [ 5193.954156] Key type ._llcrypt registered [ 5193.956597] Key type .llcrypt registered [ 5194.065642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5194.812025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5194.927909] alg: No test for adler32 (adler32-zlib) [ 5195.956605] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 5196.146849] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5197.823385] Key type lgssc registered [ 5198.995894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5205.483164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5205.495241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.920467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.747299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5216.250641] LustreError: 164868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.268554] LustreError: 164868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5219.849789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.223697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.381957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5232.881027] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5240.580214] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5248.497653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.936221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.044821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5256.559700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5264.755950] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5270.946536] Lustre: Failing over lustre-MDT0000 [ 5271.325685] Lustre: server umount lustre-MDT0000 complete [ 5271.526194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5271.543524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.572475] LustreError: 164869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.619307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.620422] LustreError: 166108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.653188] LustreError: 166108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5280.736942] LustreError: 164868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.755693] LustreError: 164868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5283.617595] LustreError: 164849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1782228052 with bad export cookie 1212149914858401411 [ 5283.635161] LustreError: 164849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5285.873781] LustreError: 164869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.890463] LustreError: 164869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5291.935431] Lustre: 164179:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782228044/real 1782228044] req@00000000ad22eb3f x1868801470656000/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782228060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5291.977103] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5292.112924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.122665] LustreError: 166108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.168748] LustreError: 166108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5301.219482] LustreError: 164177:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001b6d4e19 x1868801470661120/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5301.644804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.707634] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5306.855048] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5306.876457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5306.985719] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5309.279074] Lustre: lustre-MDT0000: Denying connection for new client 964d8830-1749-4622-98da-eda0c7fb9932 (at 192.168.204.26@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5314.450433] Lustre: lustre-MDT0000: Denying connection for new client 964d8830-1749-4622-98da-eda0c7fb9932 (at 192.168.204.26@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5321.708113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5321.715519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.733592] Lustre: Skipped 1 previous similar message [ 5321.741863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5323.952988] Lustre: server umount lustre-OST0000 complete [ 5328.007263] LustreError: 168598:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5328.020522] Lustre: 167948:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5328.026980] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5328.038453] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5328.047697] LustreError: 167948:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006082f090 x1868801470676736/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5328.060586] LustreError: 167948:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5328.067374] LustreError: 167948:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5328.074953] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5328.638144] Lustre: server umount lustre-MDT0000 complete [ 5332.453739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.477924] Lustre: Skipped 1 previous similar message [ 5332.496310] LustreError: 166844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.508668] LustreError: 166844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5332.752881] LustreError: 167564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228101 with bad export cookie 1212149914858401509 [ 5332.758897] LustreError: 164177:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5332.759306] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.772698] LustreError: 167564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5335.262620] Lustre: server umount lustre-MDT0001 complete [ 5346.785283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5349.648319] Key type lgssc unregistered [ 5350.074610] LNet: 169351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5350.096992] LNetError: 169351:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5350.116940] LNet: Removed LNI 192.168.204.126@tcp [ 5351.132462] Key type .llcrypt unregistered [ 5351.135653] Key type ._llcrypt unregistered [ 5374.616767] Key type ._llcrypt registered [ 5374.628131] Key type .llcrypt registered [ 5374.771352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5375.656930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5375.721702] alg: No test for adler32 (adler32-zlib) [ 5376.858562] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 5377.047280] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5378.727188] Key type lgssc registered [ 5379.834938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5388.231598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5388.255807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.633315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.608790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5400.047065] LustreError: 170763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.080711] LustreError: 170763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5404.175351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.758952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5410.183469] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5418.220710] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.608740] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5430.350476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5430.827143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.956828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5437.125918] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5446.492610] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5454.306195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.363929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.335169] Lustre: server umount lustre-OST0000 complete [ 5460.962094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5460.979396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.002421] Lustre: Skipped 1 previous similar message [ 5461.014596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5461.020564] Lustre: Skipped 1 previous similar message [ 5464.549987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.551054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5464.573063] Lustre: Skipped 1 previous similar message [ 5465.836543] Lustre: server umount lustre-MDT0000 complete [ 5469.667287] LustreError: 170762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5469.680609] LustreError: 170762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5469.770584] LustreError: 170742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228238 with bad export cookie 198813075636872622 [ 5469.775623] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5469.780555] LustreError: 170742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5470.081496] Lustre: server umount lustre-MDT0001 complete [ 5480.918873] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5483.606327] Key type lgssc unregistered [ 5483.919314] LNet: 174298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5483.927857] LNetError: 174298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5483.961393] LNet: Removed LNI 192.168.204.126@tcp [ 5484.815455] Key type .llcrypt unregistered [ 5484.820719] Key type ._llcrypt unregistered [ 5521.475564] Key type ._llcrypt registered [ 5521.478573] Key type .llcrypt registered [ 5521.629067] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5523.165259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5523.192605] alg: No test for adler32 (adler32-zlib) [ 5524.500950] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 5524.915400] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5526.647835] Key type lgssc registered [ 5527.975832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5536.343597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5536.376105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.861552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.461275] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5548.001419] LustreError: 176004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.030796] LustreError: 176004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5552.883512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.127839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5553.257567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.499890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5568.203411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5575.181267] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5582.718466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.326995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.163894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5601.572187] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5607.961356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5618.170663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.171976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5618.187048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.200087] Lustre: Skipped 1 previous similar message [ 5621.248147] Lustre: server umount lustre-OST0000 complete [ 5628.385302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.389710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5628.391863] Lustre: Skipped 2 previous similar messages [ 5628.416327] Lustre: Skipped 1 previous similar message [ 5631.501325] Lustre: server umount lustre-MDT0000 complete [ 5633.509713] LustreError: 176001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5633.541440] LustreError: 176001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5635.739839] LustreError: 175986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228404 with bad export cookie 6195593459850738594 [ 5635.749480] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.194641] Lustre: server umount lustre-MDT0001 complete [ 5650.883571] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5654.648955] Key type lgssc unregistered [ 5655.032576] LNet: 179553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5655.046632] LNetError: 179553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5655.086269] LNet: Removed LNI 192.168.204.126@tcp [ 5656.116530] Key type .llcrypt unregistered [ 5656.120236] Key type ._llcrypt unregistered [ 5685.409690] Key type ._llcrypt registered [ 5685.411640] Key type .llcrypt registered [ 5685.494178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5687.128450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5687.181915] alg: No test for adler32 (adler32-zlib) [ 5688.779620] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 5689.121935] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5690.876946] Key type lgssc registered [ 5692.992816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5706.530369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.554604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.000320] Lustre: *** cfs_fail_loc=135, val=0*** [ 5708.131020] LustreError: 181077:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5708.139090] LustreError: 181077:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.126@tcp: cfg command failed: rc = -2 [ 5708.154557] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5708.154557] [ 5708.165106] LustreError: MGC192.168.204.126@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5708.184498] LustreError: 181039:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5708.199505] LustreError: 181039:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5708.215719] LustreError: 181039:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5708.226895] LustreError: 181039:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5708.328167] Lustre: server umount lustre-MDT0000 complete [ 5708.330891] LustreError: 181039:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5716.335104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.736744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5722.195925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5726.188044] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.204294] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5730.781529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.184426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.088525] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5743.008179] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5750.114988] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5756.453522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.478712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.487422] Lustre: Skipped 1 previous similar message [ 5758.607303] Lustre: server umount lustre-MDT0000 complete [ 5761.511892] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.541857] LustreError: 181451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5763.312783] LustreError: 181430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228532 with bad export cookie 11133302688642018191 [ 5763.321273] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.325673] LustreError: 181430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5763.797104] Lustre: server umount lustre-MDT0001 complete [ 5776.864737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5780.259876] Key type lgssc unregistered [ 5780.681470] LNet: 183773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5780.698365] LNetError: 183773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5781.733915] LNet: Removed LNI 192.168.204.126@tcp [ 5782.732234] Key type .llcrypt unregistered [ 5782.739837] Key type ._llcrypt unregistered [ 5811.749345] Key type ._llcrypt registered [ 5811.750993] Key type .llcrypt registered [ 5811.963620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5812.321303] Key type .llcrypt unregistered [ 5812.323764] Key type ._llcrypt unregistered [ 5830.032569] Key type ._llcrypt registered [ 5830.036198] Key type .llcrypt registered [ 5830.154616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5831.449408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.533163] alg: No test for adler32 (adler32-zlib) [ 5832.657515] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 5832.861694] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5834.519266] Key type lgssc registered [ 5835.888281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5843.715420] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5843.750056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5859.299981] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5864.416512] LustreError: 185911:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 5891.103176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.575995] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5904.612226] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5910.951771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.711645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.782174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5915.848455] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5917.159452] LustreError: 185936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5917.184610] LustreError: 185936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5922.286934] LustreError: 185937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.951100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.371565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5930.276550] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5940.372665] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5946.304448] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5949.387980] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5963.241767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5963.254916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.271497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5966.310974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5966.319913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5966.346072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5969.327479] Lustre: server umount lustre-OST0000 complete [ 5976.544891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5976.559200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.587917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.590663] Lustre: Skipped 2 previous similar messages [ 5980.796281] Lustre: server umount lustre-MDT0000 complete [ 5982.689581] LustreError: 186954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.709267] LustreError: 186954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5985.177088] LustreError: 185919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228754 with bad export cookie 3050783585943594331 [ 5985.181147] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5985.191649] LustreError: 185919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5985.548161] Lustre: server umount lustre-MDT0001 complete [ 5998.431973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6002.118035] Key type lgssc unregistered [ 6002.494523] LNet: 189930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6002.520670] LNetError: 189930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6002.541415] LNet: Removed LNI 192.168.204.126@tcp [ 6003.609797] Key type .llcrypt unregistered [ 6003.612767] Key type ._llcrypt unregistered [ 6027.682512] Key type ._llcrypt registered [ 6027.687410] Key type .llcrypt registered [ 6027.784290] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6028.788386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6028.848232] alg: No test for adler32 (adler32-zlib) [ 6029.845699] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6030.050263] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6031.703160] Key type lgssc registered [ 6032.863652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6040.035419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6040.058468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.493911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.732912] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6051.819216] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.845313] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6054.393674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.759787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6059.175348] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6066.193435] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6071.620333] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6077.355994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6077.740288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.311588] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6086.981346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6090.759338] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6096.554780] Lustre: Failing over lustre-MDT0000 [ 6096.777442] Lustre: server umount lustre-MDT0000 complete [ 6097.379981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6097.381496] LustreError: 191337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6097.389472] Lustre: Skipped 2 previous similar messages [ 6097.401315] LustreError: 191337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6098.864590] LustreError: 191337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6098.885580] LustreError: 191337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6102.497981] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6102.514349] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6107.618053] LustreError: 191342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.634218] LustreError: 191342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6113.761898] Lustre: 190655:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782228866/real 1782228866] req@00000000cf8dadfe x1868802345165824/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782228882 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6113.813805] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6115.547374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6118.882872] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6118.914088] LustreError: 191343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6124.001489] LustreError: 190651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a46a24a7 x1868802345171328/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6124.377406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.446776] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6129.071714] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6129.556812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6129.652538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6129.689796] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6129.779597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6134.210804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6144.442452] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6155.146089] Lustre: server umount lustre-OST0000 complete [ 6160.353822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.356342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6160.358113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6160.364757] Lustre: Skipped 1 previous similar message [ 6165.474501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6165.486901] Lustre: Skipped 2 previous similar messages [ 6169.632931] Lustre: server umount lustre-MDT0000 complete [ 6170.594973] LustreError: 193696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.612491] LustreError: 193696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6171.679309] Lustre: 190653:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782228924/real 1782228924] req@0000000098778540 x1868802345199872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782228940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6171.707991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.208409] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782228942 with bad export cookie 7569134703493239421 [ 6173.215277] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.224298] LustreError: 191324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6173.546893] Lustre: server umount lustre-MDT0001 complete [ 6184.240635] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6186.478958] Key type lgssc unregistered [ 6186.865795] LNet: 196275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6186.873500] LNetError: 196275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6186.886181] LNet: Removed LNI 192.168.204.126@tcp [ 6187.621942] Key type .llcrypt unregistered [ 6187.627313] Key type ._llcrypt unregistered [ 6214.495725] Key type ._llcrypt registered [ 6214.499955] Key type .llcrypt registered [ 6214.606886] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6215.623862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.634200] alg: No test for adler32 (adler32-zlib) [ 6216.738449] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6216.978388] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6218.666226] Key type lgssc registered [ 6219.524625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6226.233579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6226.249334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.751761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.510548] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6237.154291] LustreError: 197690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.164111] LustreError: 197690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6238.350203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.894855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.884988] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6250.279723] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6256.124952] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6263.052534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.359968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6268.857766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6272.568135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6276.850881] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6279.884597] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6285.559456] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6302.332932] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6318.561873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.563451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6318.584921] Lustre: Skipped 1 previous similar message [ 6318.585299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6318.597297] Lustre: Skipped 1 previous similar message [ 6322.379184] Lustre: server umount lustre-OST0000 complete [ 6328.805534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.811627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6328.820362] Lustre: Skipped 1 previous similar message [ 6328.829598] Lustre: Skipped 1 previous similar message [ 6332.509927] Lustre: server umount lustre-MDT0000 complete [ 6333.929250] LustreError: 198328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.963073] LustreError: 198328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6337.342342] LustreError: 197672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782229106 with bad export cookie 8876488287729831995 [ 6337.346235] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.352606] LustreError: 197672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6337.744707] Lustre: server umount lustre-MDT0001 complete [ 6352.351736] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6355.009121] Key type lgssc unregistered [ 6355.254454] LNet: 201581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6355.262427] LNetError: 201581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6355.282474] LNet: Removed LNI 192.168.204.126@tcp [ 6356.231272] Key type .llcrypt unregistered [ 6356.232931] Key type ._llcrypt unregistered [ 6375.058237] Key type ._llcrypt registered [ 6375.060578] Key type .llcrypt registered [ 6375.183668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6376.463061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6376.480376] alg: No test for adler32 (adler32-zlib) [ 6377.741665] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6378.046026] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6379.783162] Key type lgssc registered [ 6381.397885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6389.503400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6389.548671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6391.324581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.410089] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6401.510708] LustreError: 202846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.536164] LustreError: 202846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6405.279937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.823496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6410.335801] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6418.309581] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6423.958312] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6430.808925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6431.285898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6436.400275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6437.584987] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6444.974751] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6450.204643] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6461.597718] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6468.351652] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6475.813481] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6486.500485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6486.502725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6486.516770] Lustre: Skipped 1 previous similar message [ 6486.534748] Lustre: Skipped 1 previous similar message [ 6491.623199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6491.639373] Lustre: Skipped 1 previous similar message [ 6492.495536] Lustre: server umount lustre-OST0000 complete [ 6496.737672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.750986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6496.758913] Lustre: Skipped 1 previous similar message [ 6501.856619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6501.859643] Lustre: Skipped 2 previous similar messages [ 6502.345305] Lustre: server umount lustre-MDT0000 complete [ 6506.275828] LustreError: 202826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782229275 with bad export cookie 17927326438360599056 [ 6506.280612] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6506.295546] LustreError: 202826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6506.750667] Lustre: server umount lustre-MDT0001 complete [ 6516.268204] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6518.773893] Key type lgssc unregistered [ 6519.047719] LNet: 207772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6519.057304] LNetError: 207772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6519.101944] LNet: Removed LNI 192.168.204.126@tcp [ 6519.990106] Key type .llcrypt unregistered [ 6519.992927] Key type ._llcrypt unregistered [ 6544.769190] Key type ._llcrypt registered [ 6544.771538] Key type .llcrypt registered [ 6544.891725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6546.002502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6546.106530] alg: No test for adler32 (adler32-zlib) [ 6547.298149] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6547.714213] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6549.495179] Key type lgssc registered [ 6550.727043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6560.161102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6560.195235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.864145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.954154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6572.004439] LustreError: 209187:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6572.022920] LustreError: 209187:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6576.385574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.754367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6581.387541] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6589.506747] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6595.876556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6603.083197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.412274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.559196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6609.776804] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6619.017659] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6637.545477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6637.553437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.557870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6637.557991] LustreError: Skipped 1 previous similar message [ 6637.588878] Lustre: Skipped 1 previous similar message [ 6641.964989] Lustre: server umount lustre-OST0000 complete [ 6647.783744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.786644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6647.795106] Lustre: Skipped 1 previous similar message [ 6647.807951] Lustre: Skipped 1 previous similar message [ 6652.239344] Lustre: server umount lustre-MDT0000 complete [ 6652.905145] LustreError: 211632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.936899] LustreError: 211632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6656.039492] LustreError: 209168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782229425 with bad export cookie 16677031633978733767 [ 6656.052980] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6656.467991] Lustre: server umount lustre-MDT0001 complete [ 6670.028510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6673.001329] Key type lgssc unregistered [ 6673.264592] LNet: 212873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6673.275883] LNetError: 212873:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6673.309566] LNet: Removed LNI 192.168.204.126@tcp [ 6674.366267] Key type .llcrypt unregistered [ 6674.372782] Key type ._llcrypt unregistered [ 6695.420877] Key type ._llcrypt registered [ 6695.424606] Key type .llcrypt registered [ 6695.597420] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6696.841694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6696.876168] alg: No test for adler32 (adler32-zlib) [ 6697.948842] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6698.242025] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6699.903158] Key type lgssc registered [ 6700.972847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6709.240045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6709.271140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.689434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.171241] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6720.996326] LustreError: 214145:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.012178] LustreError: 214145:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6723.431487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6723.746652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6728.060438] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6736.724769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6742.418618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6749.401621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.750588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6755.910923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6756.243948] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6764.057485] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6781.409907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.422871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.453946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6783.972616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.975137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6786.301810] Lustre: server umount lustre-OST0000 complete [ 6790.625795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6790.643908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.668436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6790.672966] Lustre: Skipped 1 previous similar message [ 6794.213213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6794.225097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.284935] Lustre: server umount lustre-MDT0000 complete [ 6799.332166] LustreError: 214139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6799.362843] LustreError: 214139:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6800.192818] LustreError: 214125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782229569 with bad export cookie 5505269084230060166 [ 6800.197966] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6800.206337] LustreError: 214125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6800.546661] Lustre: server umount lustre-MDT0001 complete [ 6811.447942] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6814.217260] Key type lgssc unregistered [ 6814.461643] LNet: 217827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6814.470162] LNetError: 217827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6814.486830] LNet: Removed LNI 192.168.204.126@tcp [ 6815.328160] Key type .llcrypt unregistered [ 6815.330854] Key type ._llcrypt unregistered [ 6835.521765] Key type ._llcrypt registered [ 6835.524941] Key type .llcrypt registered [ 6835.618108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6836.382176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6836.436196] alg: No test for adler32 (adler32-zlib) [ 6837.526660] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 6837.804601] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6839.567172] Key type lgssc registered [ 6840.837368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6848.227522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6848.260095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.684375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.725984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6859.238763] LustreError: 219093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.254804] LustreError: 219093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6862.424479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6862.865374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.239644] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6874.705823] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6881.424918] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6888.204259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6888.547415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.731032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6895.494178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6903.083860] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6913.353844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6913.506400] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6913.749565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6920.419802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6928.115267] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6929.865578] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6941.638256] Lustre: Permanently deactivating lustre-OST0001 [ 6941.650355] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6950.477516] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6959.214889] Lustre: Permanently reactivating lustre-OST0001 [ 6959.255050] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6959.262703] Lustre: Skipped 2 previous similar messages [ 6966.842558] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6966.857783] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6966.865236] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6966.874840] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6966.911821] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6970.340784] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.345936] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6970.354074] Lustre: Skipped 2 previous similar messages [ 6970.368564] Lustre: Skipped 1 previous similar message [ 6972.442157] Lustre: server umount lustre-OST0001 complete [ 6975.465084] LustreError: 220914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.499304] LustreError: 220914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6976.999142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6977.008231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.025309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6979.040941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.054852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.069863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6980.584389] LustreError: 221888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6980.618181] LustreError: 221888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6982.885420] Lustre: server umount lustre-OST0000 complete [ 6990.820841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6990.822954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.834551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6990.842777] Lustre: Skipped 1 previous similar message [ 6990.865368] Lustre: Skipped 3 previous similar messages [ 6993.402747] Lustre: server umount lustre-MDT0000 complete [ 6995.937748] LustreError: 219092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6995.959635] LustreError: 219092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6997.499628] LustreError: 219073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782229766 with bad export cookie 4851361130409991058 [ 6997.501203] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.507773] LustreError: 219073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6997.833978] Lustre: server umount lustre-MDT0001 complete [ 7009.572090] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7013.112669] Key type lgssc unregistered [ 7013.470839] LNet: 224169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7013.484302] LNetError: 224169:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7013.515888] LNet: Removed LNI 192.168.204.126@tcp [ 7015.028610] Key type .llcrypt unregistered [ 7015.032504] Key type ._llcrypt unregistered [ 7058.745850] Key type ._llcrypt registered [ 7058.747975] Key type .llcrypt registered [ 7058.882587] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7060.119411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7060.170991] alg: No test for adler32 (adler32-zlib) [ 7061.227494] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 7061.605696] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7063.319215] Key type lgssc registered [ 7064.598650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7074.397790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7074.435158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7075.733911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7075.796176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7076.066128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7081.565648] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7091.036982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.132466] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7091.171574] Lustre: 226373:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.444889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7096.847321] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7106.369440] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7114.100509] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7122.296021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7122.636203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7122.851189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7130.027230] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7132.821464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7138.087606] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7141.069466] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7149.839541] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7161.823647] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7167.167771] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7175.176042] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7196.660881] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7196.686448] Lustre: Skipped 2 previous similar messages [ 7229.528938] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7229.541583] Lustre: Skipped 3 previous similar messages [ 7304.600775] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7304.609359] Lustre: Skipped 9 previous similar messages [ 7311.842380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7311.853288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.868471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7312.354561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7312.365170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.382637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.433642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7315.453338] Lustre: Skipped 1 previous similar message [ 7317.900513] Lustre: server umount lustre-OST0000 complete [ 7322.080584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7322.093135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.117931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.665752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.003636] Lustre: server umount lustre-MDT0000 complete [ 7332.146205] LustreError: 228946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782230101 with bad export cookie 1517492241400886621 [ 7332.159467] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7332.165921] LustreError: 228946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7332.713206] Lustre: server umount lustre-MDT0001 complete [ 7344.641910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7348.238353] Key type lgssc unregistered [ 7348.596489] LNet: 231202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7348.608309] LNetError: 231202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7348.639417] LNet: Removed LNI 192.168.204.126@tcp [ 7349.515488] Key type .llcrypt unregistered [ 7349.520670] Key type ._llcrypt unregistered [ 7377.185996] Key type ._llcrypt registered [ 7377.188752] Key type .llcrypt registered [ 7377.340635] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7378.986654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.068058] alg: No test for adler32 (adler32-zlib) [ 7380.226731] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 7380.564331] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7382.255615] Key type lgssc registered [ 7383.867758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7393.256922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7393.269191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.691453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.293433] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7405.037132] LustreError: 232619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.055661] LustreError: 232619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7409.676399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7410.181162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.355416] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7424.011296] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7430.039922] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7437.969965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7438.442779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.510619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7445.138489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7453.699100] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7476.720647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7476.724158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7476.739888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7476.749532] Lustre: Skipped 1 previous similar message [ 7476.763128] Lustre: Skipped 1 previous similar message [ 7479.001205] Lustre: server umount lustre-OST0000 complete [ 7486.945583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.954500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7486.974323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7486.983043] LustreError: Skipped 1 previous similar message [ 7487.012159] Lustre: Skipped 1 previous similar message [ 7489.964765] Lustre: server umount lustre-MDT0000 complete [ 7492.068558] LustreError: 232618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.088643] LustreError: 232618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7494.289415] LustreError: 235559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782230263 with bad export cookie 10500338515376844858 [ 7494.290134] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7494.298367] LustreError: 235559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7494.744859] Lustre: server umount lustre-MDT0001 complete [ 7508.454686] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7511.881505] Key type lgssc unregistered [ 7512.206680] LNet: 236312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7512.221299] LNetError: 236312:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7512.281543] LNet: Removed LNI 192.168.204.126@tcp [ 7513.273601] Key type .llcrypt unregistered [ 7513.277456] Key type ._llcrypt unregistered [ 7539.232677] Key type ._llcrypt registered [ 7539.235345] Key type .llcrypt registered [ 7539.377851] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7540.898103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7540.915199] alg: No test for adler32 (adler32-zlib) [ 7542.099478] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 7542.402973] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7544.143287] Key type lgssc registered [ 7545.399500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7817.806504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7820.699446] Key type lgssc unregistered [ 7820.981865] LNet: 238350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.989450] LNetError: 238350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7821.005787] LNet: Removed LNI 192.168.204.126@tcp [ 7821.865239] Key type .llcrypt unregistered [ 7821.873201] Key type ._llcrypt unregistered [ 7851.016541] Key type ._llcrypt registered [ 7851.020571] Key type .llcrypt registered [ 7851.176725] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7868.339552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7869.404648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.431806] alg: No test for adler32 (adler32-zlib) [ 7870.485893] Lustre: Lustre: Build Version: 2.17.54_83_g330a3f4 [ 7870.732993] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7872.423155] Key type lgssc registered [ 7873.698042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7908.394815] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7622 sec ========== 12:04:34 (1782230674) [ 7910.284590] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:04:36 (1782230676) === [ 7914.364958] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:04:40 (1782230680) === [ 7956.261913] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7959.096621] Key type lgssc unregistered [ 7959.368246] LNet: 243587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7959.374752] LNetError: 243587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7959.406670] LNet: Removed LNI 192.168.204.126@tcp [ 7960.299151] Key type .llcrypt unregistered [ 7960.300500] Key type ._llcrypt unregistered