[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 671500289 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003391] x2apic enabled [ 0.005006] Switched APIC routing to physical x2apic. [ 0.006017] kvm-guest: setup PV IPIs [ 0.009652] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010022] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011014] pid_max: default: 32768 minimum: 301 [ 0.012144] LSM: Security Framework initializing [ 0.013060] Yama: becoming mindful. [ 0.015035] SELinux: Initializing. [ 0.016104] *** VALIDATE selinux *** [ 0.026253] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031190] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033037] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035361] *** VALIDATE tmpfs *** [ 0.037319] *** VALIDATE proc *** [ 0.038245] *** VALIDATE cgroup *** [ 0.039010] *** VALIDATE cgroup2 *** [ 0.040297] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043036] Spectre V2 : User space: Vulnerable [ 0.044012] Speculative Store Bypass: Vulnerable [ 0.047076] debug: unmapping init [mem 0xffffffff9a859000-0xffffffff9a860fff] [ 0.049939] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051025] ... version: 2 [ 0.052013] ... bit width: 48 [ 0.053014] ... generic registers: 4 [ 0.054014] ... value mask: 0000ffffffffffff [ 0.055016] ... max period: 00007fffffffffff [ 0.056015] ... fixed-purpose events: 3 [ 0.057014] ... event mask: 000000070000000f [ 0.058274] rcu: Hierarchical SRCU implementation. [ 0.060621] smp: Bringing up secondary CPUs ... [ 0.061675] x86: Booting SMP configuration: [ 0.062030] .... node #0, CPUs: #1 #2 #3 [ 0.074207] smp: Brought up 1 node, 4 CPUs [ 0.076015] smpboot: Max logical packages: 1 [ 0.077016] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.106000] node 0 deferred pages initialised in 27ms [ 0.110035] devtmpfs: initialized [ 0.111368] x86/mm: Memory block size: 128MB [ 0.114223] gcov: version magic: 0x41383552 [ 0.116177] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118318] pinctrl core: initialized pinctrl subsystem [ 0.120215] [ 0.120844] ************************************************************* [ 0.121013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122014] ** ** [ 0.123028] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124017] ** ** [ 0.125014] ** This means that this kernel is built to expose internal ** [ 0.126012] ** IOMMU data structures, which may compromise security on ** [ 0.127011] ** your system. ** [ 0.128018] ** ** [ 0.129017] ** If you see this message and you are not debugging the ** [ 0.130014] ** kernel, report this immediately to your vendor! ** [ 0.131034] ** ** [ 0.132017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133016] ************************************************************* [ 0.134789] NET: Registered protocol family 16 [ 0.135457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138747] cpuidle: using governor menu [ 0.142000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.150181] PCI: Using configuration type 1 for base access [ 0.154130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.170040] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.171022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.173175] cryptd: max_cpu_qlen set to 1000 [ 0.178429] ACPI: Added _OSI(Module Device) [ 0.179021] ACPI: Added _OSI(Processor Device) [ 0.180025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.181084] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187441] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.192538] ACPI: Interpreter enabled [ 0.193077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.194017] ACPI: Using IOAPIC for interrupt routing [ 0.195131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196492] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.208845] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.209048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.210023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.211092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.213429] acpiphp: Slot [2] registered [ 0.214485] acpiphp: Slot [5] registered [ 0.215185] acpiphp: Slot [6] registered [ 0.216220] acpiphp: Slot [7] registered [ 0.217224] acpiphp: Slot [8] registered [ 0.218225] acpiphp: Slot [9] registered [ 0.219379] acpiphp: Slot [10] registered [ 0.220633] acpiphp: Slot [3] registered [ 0.221609] acpiphp: Slot [4] registered [ 0.222147] acpiphp: Slot [11] registered [ 0.223140] acpiphp: Slot [12] registered [ 0.224135] acpiphp: Slot [13] registered [ 0.225132] acpiphp: Slot [14] registered [ 0.226229] acpiphp: Slot [15] registered [ 0.227280] acpiphp: Slot [16] registered [ 0.228121] acpiphp: Slot [17] registered [ 0.229351] acpiphp: Slot [18] registered [ 0.230137] acpiphp: Slot [19] registered [ 0.231220] acpiphp: Slot [20] registered [ 0.232115] acpiphp: Slot [21] registered [ 0.233271] acpiphp: Slot [22] registered [ 0.234125] acpiphp: Slot [23] registered [ 0.235154] acpiphp: Slot [24] registered [ 0.236125] acpiphp: Slot [25] registered [ 0.237197] acpiphp: Slot [26] registered [ 0.238263] acpiphp: Slot [27] registered [ 0.239133] acpiphp: Slot [28] registered [ 0.240233] acpiphp: Slot [29] registered [ 0.241169] acpiphp: Slot [30] registered [ 0.242173] acpiphp: Slot [31] registered [ 0.243206] PCI host bridge to bus 0000:00 [ 0.244022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.246026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.247028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.248026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.249028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.250192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252149] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.254555] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.260846] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.263679] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.264021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.265021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.266021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.267879] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.271040] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.276059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.282179] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.289015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.323391] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.338020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.348024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.381019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.407190] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.445027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.499131] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.519026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.551017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.560000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.564019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.573020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.616015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633060] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.646886] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.659018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.682024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704942] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.715827] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.719490] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.723750] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.724253] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.732381] iommu: Default domain type: Passthrough [ 0.737790] SCSI subsystem initialized [ 0.740191] ACPI: bus type USB registered [ 0.743148] usbcore: registered new interface driver usbfs [ 0.744095] usbcore: registered new interface driver hub [ 0.746100] usbcore: registered new device driver usb [ 0.749216] pps_core: LinuxPPS API ver. 1 registered [ 0.753013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.758063] PTP clock support registered [ 0.761221] EDAC MC: Ver: 3.0.0 [ 0.762088] PCI: Using ACPI for IRQ routing [ 0.764143] NetLabel: Initializing [ 0.765014] NetLabel: domain hash size = 128 [ 0.767009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769080] NetLabel: unlabeled traffic allowed by default [ 0.771376] vgaarb: loaded [ 0.774450] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776301] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784208] clocksource: Switched to clocksource kvm-clock [ 0.975105] VFS: Disk quotas dquot_6.6.0 [ 0.977786] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.980365] *** VALIDATE ramfs *** [ 0.981921] *** VALIDATE hugetlbfs *** [ 0.984282] pnp: PnP ACPI init [ 0.988329] pnp: PnP ACPI: found 6 devices [ 1.012735] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.019243] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.022299] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.025237] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.028211] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.031445] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.034942] NET: Registered protocol family 2 [ 1.038081] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.044059] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.049871] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.056194] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.061376] TCP: Hash tables configured (established 65536 bind 65536) [ 1.067396] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.074260] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.080445] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.088139] NET: Registered protocol family 1 [ 1.092467] RPC: Registered named UNIX socket transport module. [ 1.096221] RPC: Registered udp transport module. [ 1.101371] RPC: Registered tcp transport module. [ 1.104821] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.111828] NET: Registered protocol family 44 [ 1.117830] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.121923] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.124913] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.130257] PCI: CLS 0 bytes, default 64 [ 1.136381] Unpacking initramfs... [ 4.451477] debug: unmapping init [mem 0xffff9adb7cc54000-0xffff9adb7ffbffff] [ 4.464320] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.468122] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.474477] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 5.479278] Initialise system trusted keyrings [ 5.483880] Key type blacklist registered [ 5.487224] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.521927] zbud: loaded [ 5.535491] *** VALIDATE nfs *** [ 5.536726] *** VALIDATE nfs4 *** [ 5.543500] pstore: using deflate compression [ 5.561729] Platform Keyring initialized [ 5.839575] NET: Registered protocol family 38 [ 5.842262] Key type asymmetric registered [ 5.843986] Asymmetric key parser 'x509' registered [ 5.848913] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.859583] io scheduler mq-deadline registered [ 5.864985] io scheduler kyber registered [ 5.870296] io scheduler bfq registered [ 5.876349] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.887321] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.896375] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.904472] ACPI: Power Button [PWRF] [ 5.913830] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.923705] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.952393] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.961699] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.979625] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.012978] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.043131] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.049235] Non-volatile memory driver v1.3 [ 6.052220] Linux agpgart interface v0.103 [ 6.106366] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 6.109451] vda: detected capacity change from 0 to 74706944 [ 6.129508] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.132984] vdb: detected capacity change from 0 to 1073741824 [ 6.151652] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.154414] vdc: detected capacity change from 0 to 2621440000 [ 6.170251] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.174075] vdd: detected capacity change from 0 to 2621440000 [ 6.191498] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.194974] vde: detected capacity change from 0 to 4294967296 [ 6.212496] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.216708] vdf: detected capacity change from 0 to 4294967296 [ 6.224293] libphy: Fixed MDIO Bus: probed [ 6.230853] usbcore: registered new interface driver usbserial_generic [ 6.233936] usbserial: USB Serial support registered for generic [ 6.237218] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.243593] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.245895] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.248497] mousedev: PS/2 mouse device common for all mice [ 6.252205] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.253386] rtc_cmos 00:05: RTC can wake from S4 [ 6.262989] rtc_cmos 00:05: registered as rtc0 [ 6.265293] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.269957] intel_pstate: CPU model not supported [ 6.271726] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.277497] hid: raw HID events driver (C) Jiri Kosina [ 6.284947] usbcore: registered new interface driver usbhid [ 6.287265] usbhid: USB HID core driver [ 6.289097] drop_monitor: Initializing network drop monitor service [ 6.292292] Initializing XFRM netlink socket [ 6.294778] NET: Registered protocol family 10 [ 6.298427] Segment Routing with IPv6 [ 6.300825] NET: Registered protocol family 17 [ 6.302794] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.307356] mpls_gso: MPLS GSO support [ 6.313112] RAS: Correctable Errors collector initialized. [ 6.315794] AVX version of gcm_enc/dec engaged. [ 6.317656] AES CTR mode by8 optimization enabled [ 6.426240] sched_clock: Marking stable (6426149672, 0)->(7870361329, -1444211657) [ 6.430689] registered taskstats version 1 [ 6.432876] Loading compiled-in X.509 certificates [ 6.435637] zswap: loaded using pool lzo/zbud [ 6.469620] Key type big_key registered [ 6.491974] Key type encrypted registered [ 6.493824] ima: No TPM chip found, activating TPM-bypass! [ 6.496435] ima: Allocated hash algorithm: sha1 [ 6.498446] ima: No architecture policies found [ 6.500416] evm: Initialising EVM extended attributes: [ 6.502705] evm: security.selinux [ 6.504278] evm: security.ima [ 6.505571] evm: security.capability [ 6.506811] evm: HMAC attrs: 0x1 [ 6.510364] rtc_cmos 00:05: setting system clock to 2026-08-11 03:10:10 UTC (1786417810) [ 6.517726] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 6.521469] debug: unmapping init [mem 0xffffffff9a582000-0xffffffff9a858fff] [ 6.530181] Write protecting the kernel read-only data: 28672k [ 6.534196] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 6.537328] debug: unmapping init [mem 0xffffffff99514000-0xffffffff995fffff] [ 6.576593] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.585830] systemd[1]: Detected virtualization kvm. [ 6.588460] systemd[1]: Detected architecture x86-64. [ 6.590983] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.664252] systemd[1]: No hostname configured. [ 6.670697] systemd[1]: Set hostname to . [ 6.675884] random: systemd: uninitialized urandom read (16 bytes read) [ 6.684217] systemd[1]: Initializing machine ID from random generator. [ 6.828840] random: ln: uninitialized urandom read (6 bytes read) [ 7.148144] random: systemd: uninitialized urandom read (16 bytes read) [ 7.153332] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.163329] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 7.171641] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.647779] device-mapper: uevent: version 1.0.3 [ 9.652707] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.490609] virtio_net virtio0 ens2: renamed from eth0 [ 12.625600] scsi host0: ata_piix [ 12.653253] random: fast init done [ 12.789326] scsi host1: ata_piix [ 12.803207] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.830051] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.026231] random: crng init done [ 18.032413] random: 7 urandom warning(s) missed due to ratelimiting [ 20.651250] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 22.397358] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.281799] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.918888] SELinux: Disabled at runtime. [ 27.048269] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.076124] systemd[1]: Detected virtualization kvm. [ 27.087744] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.513530] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.531611] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.555849] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.576558] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.596670] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.646514] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.678652] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ 29.329743] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 31.692911] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.844649] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.971846] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.940739] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.071622] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit)[ 40.991405] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (13s / no limit)[ 41.686527] NFS: Registering the id_resolver key type [ 41.691075] Key type id_resolver registered [ 41.696732] Key type id_legacy registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 110.500030] hrtimer: interrupt took 23933243 ns [ 111.431855] libcfs: loading out-of-tree module taints kernel. [ 111.490874] Key type ._llcrypt registered [ 111.493794] Key type .llcrypt registered [ 111.607200] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 133.790490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 136.976655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 136.998425] alg: No test for adler32 (adler32-zlib) [ 138.824583] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 140.074437] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 142.103193] Key type lgssc registered [ 144.448173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 168.714520] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 219.321715] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 238.259275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 238.325329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.815199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 239.863367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 240.016994] Lustre: lustre-MDT0000: new disk, initializing [ 240.129992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 240.151154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 246.596846] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.902623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.069679] Lustre: 6536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 263.109966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 263.115187] Lustre: Skipped 1 previous similar message [ 263.204973] Lustre: lustre-MDT0001: new disk, initializing [ 263.286191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 263.316673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 263.324436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 268.778110] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.087509] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 286.564974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.097593] Lustre: lustre-OST0000: new disk, initializing [ 287.105420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 287.119752] Lustre: 8475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.248107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 292.436598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 292.445595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 292.601456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 294.352947] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 312.292442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 312.468602] Lustre: lustre-OST0001: new disk, initializing [ 312.473193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 312.479692] Lustre: 9548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 312.580694] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 319.122623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 319.143658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 319.234632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 320.494536] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 334.866916] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 346.464245] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 355.355127] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 363.323794] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 371.431834] Lustre: DEBUG MARKER: Client: 2.17.56.52 [ 374.979612] Lustre: DEBUG MARKER: MDS: 2.17.56.52 [ 379.257828] Lustre: DEBUG MARKER: OSS: 2.17.56.52 [ 382.071932] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 10 23:16:24 EDT 2026 [ 405.708625] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 408.321855] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 426.975557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 426.977521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.978785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.016049] Lustre: Skipped 2 previous similar messages [ 432.116356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 432.139750] Lustre: Skipped 6 previous similar messages [ 432.468309] Lustre: server umount lustre-MDT0000 complete [ 436.714631] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418240 with bad export cookie 21395721037747048 [ 436.722858] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.733673] LustreError: 6526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 437.217161] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.218547] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 437.237982] Lustre: Skipped 2 previous similar messages [ 437.251863] Lustre: Skipped 1 previous similar message [ 441.250306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 441.257305] Lustre: Skipped 1 previous similar message [ 443.373670] Lustre: server umount lustre-MDT0001 complete [ 447.460722] Lustre: server umount lustre-OST0000 complete [ 451.444861] Lustre: server umount lustre-OST0001 complete [ 459.673861] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 471.692490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 513.359868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 513.831456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 513.885583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 514.014719] Lustre: lustre-MDT0000: new disk, initializing [ 514.200176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.247406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 520.395430] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 533.463803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.660847] Lustre: 15113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 533.752832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 533.761841] Lustre: Skipped 1 previous similar message [ 533.847972] Lustre: lustre-MDT0001: new disk, initializing [ 533.964329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 534.023264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 534.046452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 540.241432] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 552.013980] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.621626] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.249470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.716763] Lustre: lustre-OST0000: new disk, initializing [ 567.720868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 567.726561] Lustre: 17083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 567.846211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.896109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 569.912774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 570.116845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 576.021442] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 586.858187] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 592.632436] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 593.111791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 600.050667] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 600.489771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 606.183200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 606.189032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 606.217427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 606.236175] Lustre: Skipped 1 previous similar message [ 609.431442] Lustre: server umount lustre-OST0000 complete [ 615.906376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 615.912293] LustreError: Skipped 1 previous similar message [ 615.915426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.924313] Lustre: Skipped 1 previous similar message [ 615.933060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 620.407984] Lustre: server umount lustre-MDT0000 complete [ 621.545151] LustreError: 15124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.570930] LustreError: 15124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 624.807832] LustreError: 15106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418428 with bad export cookie 21395721037748581 [ 624.825442] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.145964] Lustre: server umount lustre-MDT0001 complete [ 629.201233] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 23:20:31 (1786418431) [ 637.406048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.778185] LustreError: 19349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.856819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.878512] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 643.041955] LustreError: 19350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.148444] LustreError: 19349:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.256575] LustreError: 19350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.895372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.968565] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 665.797900] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 671.297650] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 678.777880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.886354] LustreError: 21172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.078490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 694.318883] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 702.951091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.975163] Lustre: Skipped 1 previous similar message [ 702.981499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.981881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 703.002298] Lustre: Skipped 3 previous similar messages [ 707.638397] Lustre: server umount lustre-OST0000 complete [ 713.185386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 713.203016] Lustre: Skipped 2 previous similar messages [ 718.116067] Lustre: server umount lustre-MDT0000 complete [ 718.308034] LustreError: 19344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 718.320133] LustreError: 19344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 722.840184] LustreError: 19329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418526 with bad export cookie 21395721037749463 [ 722.872490] LustreError: 19329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 722.886936] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.380343] Lustre: server umount lustre-MDT0001 complete [ 740.783664] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 745.007372] Key type lgssc unregistered [ 745.456182] LNet: 22888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.467752] LNetError: 22888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.493608] LNet: Removed LNI 192.168.204.106@tcp [ 746.982628] Key type .llcrypt unregistered [ 746.986806] Key type ._llcrypt unregistered [ 777.058793] Key type ._llcrypt registered [ 777.064298] Key type .llcrypt registered [ 777.269284] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 778.681156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.725384] alg: No test for adler32 (adler32-zlib) [ 779.761752] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 779.997530] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 781.687189] Key type lgssc registered [ 783.156136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.616718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 791.637270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.047135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.925718] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 803.306505] LustreError: 24304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 803.380930] LustreError: 24304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 808.418046] LustreError: 24303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.711902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.053441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 814.995490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 825.254716] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 833.035620] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 841.617245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 841.992857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.040891] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 860.578658] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 863.633991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 883.621070] Lustre: server umount lustre-OST0000 complete [ 889.326646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.333967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 889.365822] Lustre: Skipped 1 previous similar message [ 894.438870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.455175] Lustre: Skipped 2 previous similar messages [ 898.664463] Lustre: server umount lustre-MDT0000 complete [ 899.562883] LustreError: 25737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.575290] Lustre: 23615:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786418688/real 1786418688] req@0000000084d2fd7e x1873195652121472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786418704 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 900.616217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.315809] LustreError: 24283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418707 with bad export cookie 13526245757940392769 [ 903.335365] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 903.339251] LustreError: 24283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 903.853472] Lustre: server umount lustre-MDT0001 complete [ 917.701176] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 921.142972] Key type lgssc unregistered [ 921.456738] LNet: 28287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.466126] LNetError: 28287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.493203] LNet: Removed LNI 192.168.204.106@tcp [ 922.472796] Key type .llcrypt unregistered [ 922.474911] Key type ._llcrypt unregistered [ 954.247221] Key type ._llcrypt registered [ 954.251687] Key type .llcrypt registered [ 954.491175] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 956.500995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.624553] alg: No test for adler32 (adler32-zlib) [ 958.024750] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 958.393604] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 960.103221] Key type lgssc registered [ 962.313060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.274734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.293995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.809240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.306442] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 983.010582] LustreError: 29703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.045789] LustreError: 29703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 988.134521] LustreError: 29704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 989.433968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.719927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 994.466327] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1004.248825] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1012.494329] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1030.404416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1030.866940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.965947] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1042.244917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1049.315227] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1060.836870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1060.851313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.870299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.891642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1062.892310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1062.898554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1062.911434] Lustre: Skipped 1 previous similar message [ 1064.223644] Lustre: server umount lustre-OST0000 complete [ 1071.018137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.051256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.061339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1073.081850] Lustre: Skipped 2 previous similar messages [ 1074.745942] Lustre: server umount lustre-MDT0000 complete [ 1078.245821] LustreError: 29704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.274517] LustreError: 29704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1079.726531] LustreError: 29684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786418883 with bad export cookie 11455965553312843305 [ 1079.741264] LustreError: 29684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1079.745663] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.191170] Lustre: server umount lustre-MDT0001 complete [ 1093.818899] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1097.389721] Key type lgssc unregistered [ 1097.804644] LNet: 33588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.819938] LNetError: 33588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.835596] LNet: Removed LNI 192.168.204.106@tcp [ 1099.130223] Key type .llcrypt unregistered [ 1099.132470] Key type ._llcrypt unregistered [ 1132.890663] Key type ._llcrypt registered [ 1132.893358] Key type .llcrypt registered [ 1132.994638] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1134.897930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1134.984063] alg: No test for adler32 (adler32-zlib) [ 1136.123435] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 1136.388414] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1138.151489] Key type lgssc registered [ 1140.056380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.804719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.832274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.451739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.213216] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1161.696482] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.722480] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1166.817796] LustreError: 35005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.125085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.616575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1173.055989] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1183.192933] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1191.162824] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1198.612179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1198.919390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.857401] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1216.665718] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1221.501459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1236.147135] Lustre: server umount lustre-OST0000 complete [ 1242.081121] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.094689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.109899] Lustre: Skipped 1 previous similar message [ 1242.141551] Lustre: Skipped 1 previous similar message [ 1247.201514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.211388] Lustre: Skipped 1 previous similar message [ 1250.402360] Lustre: server umount lustre-MDT0000 complete [ 1252.261333] Lustre: 34317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419040/real 1786419040] req@00000000c885a79a x1873196026462208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786419056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1252.308981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.323331] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.378060] LustreError: 35004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1253.343126] Lustre: 34316:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786419041/real 1786419041] req@0000000099b20ba8 x1873196026462592/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786419057 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1255.386595] LustreError: 34986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419059 with bad export cookie 12503708612268308988 [ 1255.393665] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.420930] LustreError: 34986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1255.964224] Lustre: server umount lustre-MDT0001 complete [ 1270.000535] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1273.574533] Key type lgssc unregistered [ 1274.043424] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.056778] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.086661] LNet: Removed LNI 192.168.204.106@tcp [ 1275.044419] Key type .llcrypt unregistered [ 1275.048491] Key type ._llcrypt unregistered [ 1303.365989] Key type ._llcrypt registered [ 1303.368288] Key type .llcrypt registered [ 1303.492552] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1304.820285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1304.859571] alg: No test for adler32 (adler32-zlib) [ 1306.170259] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 1306.461925] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1308.223215] Key type lgssc registered [ 1309.918697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.660959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1318.712953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.255345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.496749] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1330.658609] LustreError: 39977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.681976] LustreError: 39977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1334.168181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.606428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1340.264622] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1348.085539] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1354.871293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1361.356307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1361.653168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.777944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1367.686463] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1376.239497] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1387.488412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1387.495682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.508732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.223579] Lustre: lustre-OST0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1389.537171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.548944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.479886] Lustre: server umount lustre-OST0000 complete [ 1415.145128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.150336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.159251] Lustre: Skipped 1 previous similar message [ 1415.175487] Lustre: Skipped 3 previous similar messages [ 1418.746881] Lustre: server umount lustre-MDT0000 complete [ 1420.263220] LustreError: 39976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.282211] LustreError: 39976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1422.162989] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419226 with bad export cookie 1497474179949708359 [ 1422.170187] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.198869] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1428.753441] Lustre: server umount lustre-MDT0001 complete [ 1442.837454] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1446.641954] Key type lgssc unregistered [ 1446.965146] LNet: 43686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.975370] LNetError: 43686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.043517] LNet: Removed LNI 192.168.204.106@tcp [ 1448.959629] Key type .llcrypt unregistered [ 1448.962294] Key type ._llcrypt unregistered [ 1474.384705] Key type ._llcrypt registered [ 1474.387503] Key type .llcrypt registered [ 1474.478781] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1475.856435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.919755] alg: No test for adler32 (adler32-zlib) [ 1476.956264] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 1477.257291] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1478.935261] Key type lgssc registered [ 1480.298774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1488.429834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1488.465786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.917688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.724764] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1499.107185] LustreError: 45098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.128712] LustreError: 45098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1504.134185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.230172] LustreError: 45099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.496954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.893894] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1517.662363] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1524.040145] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1530.926951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1531.240855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1534.327720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1537.436837] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1545.895337] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1558.715507] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.6@tcp (stopping) [ 1560.039878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.060264] Lustre: Skipped 2 previous similar messages [ 1560.065655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.609708] Lustre: server umount lustre-MDT0000 complete [ 1563.787531] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.637281] LustreError: 45079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419370 with bad export cookie 14142440098357538890 [ 1566.652632] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1566.658427] LustreError: 45079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1568.899281] LustreError: 45093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.926205] LustreError: 45093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1568.957109] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.6@tcp (stopping) [ 1568.970773] Lustre: Skipped 2 previous similar messages [ 1570.276620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.297614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1571.305259] Lustre: Skipped 1 previous similar message [ 1573.119435] Lustre: server umount lustre-MDT0001 complete [ 1588.194308] Lustre: server umount lustre-OST0000 complete [ 1607.035455] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1610.380376] Key type lgssc unregistered [ 1610.673105] LNet: 48967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.690692] LNetError: 48967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.750568] LNet: Removed LNI 192.168.204.106@tcp [ 1613.231987] Key type .llcrypt unregistered [ 1613.237434] Key type ._llcrypt unregistered [ 1638.836949] Key type ._llcrypt registered [ 1638.841175] Key type .llcrypt registered [ 1638.946376] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1640.055635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.069796] alg: No test for adler32 (adler32-zlib) [ 1641.220914] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 1641.526461] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1643.239177] Key type lgssc registered [ 1644.691154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1652.745753] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1652.762300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.384553] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1673.503817] LustreError: 50354:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 1700.127235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.190587] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1715.144721] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1881.620411] Lustre: server umount lustre-OST0000 complete [ 1900.342561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1903.457797] Key type lgssc unregistered [ 1903.712433] LNet: 52033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.723241] LNetError: 52033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.740794] LNet: Removed LNI 192.168.204.106@tcp [ 1905.576630] Key type .llcrypt unregistered [ 1905.580222] Key type ._llcrypt unregistered [ 1930.660521] Key type ._llcrypt registered [ 1930.663253] Key type .llcrypt registered [ 1930.786385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1932.693869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1932.753126] alg: No test for adler32 (adler32-zlib) [ 1933.872743] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 1934.139222] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1935.895748] Key type lgssc registered [ 1937.329102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1945.221538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1945.236097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1946.658725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.275274] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1956.835946] LustreError: 53450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.854017] LustreError: 53450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1959.691867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.985510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1965.231402] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1973.603648] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.690374] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.493576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.853420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.984923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1992.753839] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2001.318938] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2004.904900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2004.911814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.089827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.101830] Lustre: Skipped 2 previous similar messages [ 2015.201040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2018.271496] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2018.466502] Lustre: server umount lustre-OST0000 complete [ 2025.444287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.456699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.458256] Lustre: Skipped 1 previous similar message [ 2025.492181] Lustre: Skipped 2 previous similar messages [ 2028.731556] Lustre: server umount lustre-MDT0000 complete [ 2030.563441] LustreError: 53451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.580078] LustreError: 53451:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2032.810041] LustreError: 53432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786419836 with bad export cookie 9372986098836791657 [ 2032.844452] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.137609] Lustre: server umount lustre-MDT0001 complete [ 2046.496776] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2049.567988] Key type lgssc unregistered [ 2050.009305] LNet: 56986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.029624] LNetError: 56986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.048711] LNet: Removed LNI 192.168.204.106@tcp [ 2051.298267] Key type .llcrypt unregistered [ 2051.306956] Key type ._llcrypt unregistered [ 2079.764939] Key type ._llcrypt registered [ 2079.767441] Key type .llcrypt registered [ 2079.908931] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2081.225170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2081.435381] alg: No test for adler32 (adler32-zlib) [ 2082.566883] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 2082.768768] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2084.463386] Key type lgssc registered [ 2085.615341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2093.420415] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2093.435336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2108.959419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2114.079389] LustreError: 58382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 2140.704974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.393732] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2156.940611] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2165.070405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.684910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.806891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2170.745286] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2170.852237] LustreError: 58408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2170.872297] LustreError: 58408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2175.970720] LustreError: 58407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.999102] LustreError: 58407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2180.647846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.040145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.083259] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2196.694786] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2203.658350] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2206.690434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2206.695347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.722386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2206.729755] Lustre: Skipped 1 previous similar message [ 2211.824064] Lustre: server umount lustre-OST0000 complete [ 2236.385471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.394368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.408025] Lustre: Skipped 1 previous similar message [ 2236.427884] Lustre: Skipped 2 previous similar messages [ 2239.653749] Lustre: server umount lustre-MDT0000 complete [ 2241.513099] LustreError: 59423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.530500] LustreError: 59423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2244.189769] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420048 with bad export cookie 8791023021867968234 [ 2244.197195] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2244.198960] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2250.549621] Lustre: server umount lustre-MDT0001 complete [ 2259.859904] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2262.295657] Key type lgssc unregistered [ 2262.673601] LNet: 62132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.681809] LNetError: 62132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.705583] LNet: Removed LNI 192.168.204.106@tcp [ 2263.551892] Key type .llcrypt unregistered [ 2263.564194] Key type ._llcrypt unregistered [ 2288.644868] Key type ._llcrypt registered [ 2288.647480] Key type .llcrypt registered [ 2288.748808] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2290.086644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2290.138900] alg: No test for adler32 (adler32-zlib) [ 2291.292618] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 2291.622283] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2293.415169] Key type lgssc registered [ 2294.542173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2302.740834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2302.774162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.288738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.726811] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2314.724977] LustreError: 63546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.768716] LustreError: 63546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2318.349224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.633691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.534917] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2333.769280] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.704253] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2346.663266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.968465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2349.049711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2353.580160] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2361.038465] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2379.751381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2379.757559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2379.779790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2381.130306] Lustre: server umount lustre-OST0000 complete [ 2385.379757] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2385.393937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.417254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.962401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.967167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.016610] Lustre: Skipped 1 previous similar message [ 2391.360604] Lustre: server umount lustre-MDT0000 complete [ 2394.086786] LustreError: 65180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2394.118889] LustreError: 65180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2395.560628] LustreError: 63524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420199 with bad export cookie 4234427494751887784 [ 2395.572586] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2395.583664] LustreError: 63524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2406.059748] Lustre: server umount lustre-MDT0001 complete [ 2418.382363] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2421.655565] Key type lgssc unregistered [ 2422.045853] LNet: 67106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.050783] LNetError: 67106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.061978] LNet: Removed LNI 192.168.204.106@tcp [ 2422.965349] Key type .llcrypt unregistered [ 2422.967689] Key type ._llcrypt unregistered [ 2453.459719] Key type ._llcrypt registered [ 2453.466796] Key type .llcrypt registered [ 2453.628031] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2454.623786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.687959] alg: No test for adler32 (adler32-zlib) [ 2456.337637] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 2456.710731] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2458.399232] Key type lgssc registered [ 2460.367692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2472.797616] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 23:51:15 (1786420275) [ 2480.112231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2480.138366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.558233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.260411] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2491.879084] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.917931] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2492.901109] LustreError: 68812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.155892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.486683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.377920] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2507.583909] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2513.515759] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2520.023321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.356641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.464912] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2527.553782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2535.829900] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2543.316461] Lustre: Failing over lustre-MDT0000 [ 2543.626342] Lustre: server umount lustre-MDT0000 complete [ 2543.799955] LustreError: 68808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.680898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.702785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.737072] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.646255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.753351] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.899663] Lustre: *** cfs_fail_loc=135, val=0*** [ 2552.976514] LustreError: 71900:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2552.980892] LustreError: 71900:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -2 [ 2552.986853] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2552.986853] [ 2553.004714] LustreError: MGC192.168.204.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2553.024319] LustreError: 71888:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2553.046288] LustreError: 71888:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2553.072941] LustreError: 71888:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2553.105541] LustreError: 71888:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2553.178666] Lustre: server umount lustre-MDT0000 complete [ 2553.182129] LustreError: 71888:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2553.315043] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.353038] LustreError: 68813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2559.803253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.947909] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.160656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.273359] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2564.228054] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2565.518758] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2565.630756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2565.659835] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2565.726631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2575.845521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2575.848720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2575.864675] Lustre: Skipped 3 previous similar messages [ 2579.838657] Lustre: server umount lustre-OST0000 complete [ 2586.080624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.099737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2586.108073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.132020] Lustre: Skipped 2 previous similar messages [ 2591.203022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.210581] Lustre: Skipped 1 previous similar message [ 2595.298494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.301439] Lustre: Skipped 1 previous similar message [ 2598.367422] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2598.538046] Lustre: server umount lustre-MDT0000 complete [ 2600.423984] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.448460] LustreError: 68807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2603.045759] LustreError: 68794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420407 with bad export cookie 1639507407256719997 [ 2603.051984] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.060262] LustreError: 68794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2603.614218] Lustre: server umount lustre-MDT0001 complete [ 2615.934708] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2618.797308] Key type lgssc unregistered [ 2619.116957] LNet: 73685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.128538] LNetError: 73685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.148696] LNet: Removed LNI 192.168.204.106@tcp [ 2619.899612] Key type .llcrypt unregistered [ 2619.904128] Key type ._llcrypt unregistered [ 2643.698196] Key type ._llcrypt registered [ 2643.700189] Key type .llcrypt registered [ 2643.806606] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2645.471908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.508532] alg: No test for adler32 (adler32-zlib) [ 2646.674527] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 2646.952420] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2648.607331] Key type lgssc registered [ 2649.647729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.383255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2656.413099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.772141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2661.485910] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2668.013160] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.031388] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2669.206868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.598114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.438633] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2682.304102] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2688.537995] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2697.999348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.505853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.693157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2705.950109] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2714.869538] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2721.237593] Lustre: Failing over lustre-MDT0000 [ 2721.463485] Lustre: server umount lustre-MDT0000 complete [ 2722.477924] LustreError: 75093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.351410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.356431] LustreError: 77158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2725.366609] Lustre: Skipped 2 previous similar messages [ 2725.381857] LustreError: 77158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2727.590774] LustreError: 75094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.937615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.034537] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.169410] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2732.298682] LustreError: 78185:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2732.303680] LustreError: 78185:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -9 [ 2732.308733] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2732.308733] [ 2732.314677] LustreError: MGC192.168.204.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2732.325249] LustreError: 78173:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2732.333363] LustreError: 78173:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2732.342826] LustreError: 78173:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2732.355219] LustreError: 78173:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2732.395671] Lustre: server umount lustre-MDT0000 complete [ 2732.400128] LustreError: 78173:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2732.681671] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.692315] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2740.047347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.176409] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2740.185666] LustreError: 78533:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.106@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000954029cd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2740.201308] LustreError: 75090:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2740.574169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.628573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2742.927509] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2745.847651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2745.891363] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2745.958454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2746.482173] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2756.070073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2756.075318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.078595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2756.101033] Lustre: Skipped 1 previous similar message [ 2761.124250] Lustre: server umount lustre-OST0000 complete [ 2766.305443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.309538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2766.324149] Lustre: Skipped 1 previous similar message [ 2766.338229] Lustre: Skipped 2 previous similar messages [ 2771.320684] Lustre: server umount lustre-MDT0000 complete [ 2771.424943] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.451993] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2774.868491] LustreError: 75078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420578 with bad export cookie 18143960751553132787 [ 2774.875841] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2774.882043] LustreError: 75078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2775.199648] Lustre: server umount lustre-MDT0001 complete [ 2787.481420] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2791.244930] Key type lgssc unregistered [ 2791.738891] LNet: 79974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2791.756611] LNetError: 79974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2792.810133] LNet: Removed LNI 192.168.204.106@tcp [ 2793.954949] Key type .llcrypt unregistered [ 2793.963542] Key type ._llcrypt unregistered [ 2809.307973] Key type ._llcrypt registered [ 2809.312280] Key type .llcrypt registered [ 2809.479455] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2810.010478] Key type .llcrypt unregistered [ 2810.014910] Key type ._llcrypt unregistered [ 2840.386610] Key type ._llcrypt registered [ 2840.396377] Key type .llcrypt registered [ 2840.672312] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2843.079257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2843.176895] alg: No test for adler32 (adler32-zlib) [ 2844.332825] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 2844.683753] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2846.527204] Key type lgssc registered [ 2847.866821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2857.519510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2857.569430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.218408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.050860] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2869.736363] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.785861] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2874.691248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.855463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2875.038677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2880.617779] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2890.020655] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2897.353351] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2907.247561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.635296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.172492] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2924.330138] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2929.156543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2939.369933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.370160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.402917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.410528] Lustre: Skipped 1 previous similar message [ 2944.480639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.502526] Lustre: Skipped 1 previous similar message [ 2944.909943] Lustre: server umount lustre-OST0000 complete [ 2949.604398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.630880] Lustre: Skipped 1 previous similar message [ 2949.641191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.653488] Lustre: Skipped 1 previous similar message [ 2954.717122] Lustre: server umount lustre-MDT0000 complete [ 2954.721872] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.744122] LustreError: 81880:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2958.961899] LustreError: 81861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420762 with bad export cookie 11650515088593349049 [ 2958.966342] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2958.975598] LustreError: 81861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2959.285098] Lustre: server umount lustre-MDT0001 complete [ 2971.075196] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2974.333235] Key type lgssc unregistered [ 2974.674296] LNet: 85436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2974.680765] LNetError: 85436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2974.707596] LNet: Removed LNI 192.168.204.106@tcp [ 2975.721505] Key type .llcrypt unregistered [ 2975.723979] Key type ._llcrypt unregistered [ 3003.338810] Key type ._llcrypt registered [ 3003.342117] Key type .llcrypt registered [ 3003.474500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3004.677642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3004.911893] alg: No test for adler32 (adler32-zlib) [ 3006.167165] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 3006.429706] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3008.175266] Key type lgssc registered [ 3009.517430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3018.355874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3018.384206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.869471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.249956] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3029.999498] LustreError: 86850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.031658] LustreError: 86850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3035.117648] LustreError: 86851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3037.046045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.387119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3043.571687] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3052.688242] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3061.785224] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3069.618842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.040200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.736459] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3087.016781] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.546632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3098.081720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3098.089952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.113426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.133446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.141987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.165445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3104.169192] Lustre: server umount lustre-OST0000 complete [ 3110.376989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.386171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3110.404177] Lustre: Skipped 1 previous similar message [ 3110.441682] Lustre: Skipped 2 previous similar messages [ 3115.492437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.513977] Lustre: Skipped 1 previous similar message [ 3116.761949] Lustre: server umount lustre-MDT0000 complete [ 3120.608710] LustreError: 86850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.630949] LustreError: 86850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3121.286839] LustreError: 86831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786420925 with bad export cookie 15377468716554268524 [ 3121.306602] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3121.307835] LustreError: 86831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3121.651313] Lustre: server umount lustre-MDT0001 complete [ 3136.164331] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3139.972758] Key type lgssc unregistered [ 3140.372846] LNet: 90402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3140.379977] LNetError: 90402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3140.434450] LNet: Removed LNI 192.168.204.106@tcp [ 3141.812781] Key type .llcrypt unregistered [ 3141.816410] Key type ._llcrypt unregistered [ 3173.689520] Key type ._llcrypt registered [ 3173.694949] Key type .llcrypt registered [ 3173.920630] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3175.418423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.477289] alg: No test for adler32 (adler32-zlib) [ 3176.830041] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 3177.103710] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3178.799200] Key type lgssc registered [ 3179.952215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3188.503437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3188.519637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.146188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.013720] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3200.497592] LustreError: 91818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.527676] LustreError: 91818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3205.614304] LustreError: 91819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.654806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.237329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.174332] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3221.372906] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3228.487836] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.529272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.768699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3245.130629] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3254.631672] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3260.395922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3272.674958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3272.686900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.704443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.214212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3274.226651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.245666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.753538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.845757] Lustre: server umount lustre-OST0000 complete [ 3283.937776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3283.953061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.974275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.981073] Lustre: Skipped 1 previous similar message [ 3285.988076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.108089] LustreError: 91818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.127020] LustreError: 91818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3291.625449] Lustre: server umount lustre-MDT0000 complete [ 3296.227715] LustreError: 93260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.258355] LustreError: 93260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3297.797525] LustreError: 91798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421101 with bad export cookie 1791577949369891389 [ 3297.801638] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3297.818969] LustreError: 91798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3298.125205] Lustre: server umount lustre-MDT0001 complete [ 3310.492331] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3314.314847] Key type lgssc unregistered [ 3314.623517] LNet: 95386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3314.632887] LNetError: 95386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.652554] LNet: Removed LNI 192.168.204.106@tcp [ 3315.649198] Key type .llcrypt unregistered [ 3315.652163] Key type ._llcrypt unregistered [ 3346.415570] Key type ._llcrypt registered [ 3346.422701] Key type .llcrypt registered [ 3346.610133] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3348.513594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3348.760188] alg: No test for adler32 (adler32-zlib) [ 3350.100941] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 3350.569905] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3352.303486] Key type lgssc registered [ 3353.439856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3363.369925] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3363.395472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.976083] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3384.095399] LustreError: 96774:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 3410.847209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.658565] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3429.181793] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3437.961420] Lustre: server umount lustre-OST0000 complete [ 3449.182735] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 00:07:31 (1786421251) [ 3457.087681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.393454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.625106] LustreError: 98416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.640960] LustreError: 98416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3463.183250] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3467.748974] LustreError: 98417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.866412] LustreError: 98416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.855645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.251235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3479.435866] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3487.634579] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3494.353249] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3504.204497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3504.639567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.223443] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3520.082253] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3527.175497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3537.382283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.384776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.385254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.402417] Lustre: Skipped 1 previous similar message [ 3542.497983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.508202] Lustre: Skipped 2 previous similar messages [ 3543.334734] Lustre: server umount lustre-OST0000 complete [ 3551.202589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3551.216940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.242397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.749270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.934204] Lustre: server umount lustre-MDT0000 complete [ 3557.876774] LustreError: 100891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.915081] LustreError: 100891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3558.261953] LustreError: 98397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421362 with bad export cookie 2161185847880777483 [ 3558.277121] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3558.278196] LustreError: 98397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3558.583499] Lustre: server umount lustre-MDT0001 complete [ 3570.736890] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3573.726717] Key type lgssc unregistered [ 3574.074085] LNet: 102114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3574.085793] LNetError: 102114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3574.101672] LNet: Removed LNI 192.168.204.106@tcp [ 3574.798509] Key type .llcrypt unregistered [ 3574.802107] Key type ._llcrypt unregistered [ 3624.956804] Key type ._llcrypt registered [ 3624.958991] Key type .llcrypt registered [ 3625.054923] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3626.409419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3626.490606] alg: No test for adler32 (adler32-zlib) [ 3627.609701] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 3627.895735] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3629.575255] Key type lgssc registered [ 3630.698807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3639.980525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3639.996417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.412798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.790763] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3651.561645] LustreError: 104307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3651.584995] LustreError: 104307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3656.479622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.679828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3656.836121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3662.789173] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3673.042175] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3679.613791] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3688.230452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3688.728224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.257611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3706.220941] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3711.298606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3721.703920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3721.708347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.716931] LustreError: Skipped 1 previous similar message [ 3721.738260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.765699] Lustre: Skipped 1 previous similar message [ 3726.564463] Lustre: server umount lustre-OST0000 complete [ 3731.937118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.952727] Lustre: Skipped 1 previous similar message [ 3731.962862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3731.972181] Lustre: Skipped 1 previous similar message [ 3737.056980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3737.077770] Lustre: Skipped 1 previous similar message [ 3737.591865] Lustre: server umount lustre-MDT0000 complete [ 3741.164458] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.211323] LustreError: 104306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3742.098063] LustreError: 104287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421546 with bad export cookie 10650410903892015287 [ 3742.108553] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3742.114358] LustreError: 104287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3742.456605] Lustre: server umount lustre-MDT0001 complete [ 3756.436927] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3760.194820] Key type lgssc unregistered [ 3760.667353] LNet: 107866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3760.687777] LNetError: 107866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3760.711693] LNet: Removed LNI 192.168.204.106@tcp [ 3762.014604] Key type .llcrypt unregistered [ 3762.020092] Key type ._llcrypt unregistered [ 3784.860835] Key type ._llcrypt registered [ 3784.863656] Key type .llcrypt registered [ 3785.050743] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3787.222640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3787.330075] alg: No test for adler32 (adler32-zlib) [ 3788.447309] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 3788.766757] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3790.455412] Key type lgssc registered [ 3792.407926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3801.219624] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3801.237395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3816.735508] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3821.855461] LustreError: 109109:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 3848.543554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.965287] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3864.622806] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3872.051291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3872.279294] LustreError: MGC192.168.204.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3872.288756] LustreError: 110132:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3872.294657] LustreError: 110132:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3872.301295] LustreError: 110132:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3872.479238] Lustre: server umount lustre-MDT0000 complete [ 3872.487841] LustreError: 110132:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3893.729450] Lustre: 108446:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786421681/real 1786421681] req@0000000016ef6ff8 x1873198807256448/t0(0) o400->MGC192.168.204.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786421697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3893.767975] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3893.786913] LustreError: 109121:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3914.432788] Lustre: server umount lustre-OST0000 complete [ 3926.149443] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3937.191372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3980.353788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.914525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3980.986619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3981.158876] Lustre: lustre-MDT0000: new disk, initializing [ 3981.350878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3981.381290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3987.899791] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3999.548825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.688081] Lustre: 112931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3999.778628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3999.785534] Lustre: Skipped 1 previous similar message [ 3999.870878] Lustre: lustre-MDT0001: new disk, initializing [ 4000.010277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4000.072490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4000.092085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4005.530263] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4016.087475] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.481607] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4030.130610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4030.379754] Lustre: lustre-OST0000: new disk, initializing [ 4030.384259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4030.391264] Lustre: 114900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4030.486992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.592922] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4039.307380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4039.323061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4039.433452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.870581] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.048051] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4052.471614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4057.593198] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4057.939594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4065.252595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.253704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.255816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4065.262310] Lustre: Skipped 1 previous similar message [ 4066.445622] Lustre: server umount lustre-OST0000 complete [ 4071.907411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4071.919411] LustreError: Skipped 1 previous similar message [ 4071.925410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.949607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.965895] Lustre: Skipped 1 previous similar message [ 4075.488716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.492506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4075.526360] Lustre: Skipped 1 previous similar message [ 4076.831481] Lustre: server umount lustre-MDT0000 complete [ 4081.923889] LustreError: 113733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786421885 with bad export cookie 15169743551622248372 [ 4081.929637] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4081.948879] LustreError: 113733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4082.441824] Lustre: server umount lustre-MDT0001 complete [ 4092.018541] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 00:18:14 (1786421894) [ 4096.309758] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4116.852879] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4128.067210] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4171.339015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.826692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4171.883140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4171.984784] Lustre: lustre-MDT0000: new disk, initializing [ 4172.072966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.091467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4177.643785] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4188.668626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4188.888246] Lustre: 119803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4188.992095] LustreError: 119813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4189.005130] LustreError: 119813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4189.032235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4189.036220] Lustre: Skipped 1 previous similar message [ 4189.199818] Lustre: lustre-MDT0001: new disk, initializing [ 4189.396000] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4189.606357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4189.640696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4195.286500] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4207.051814] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4213.846401] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4222.846973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.293678] Lustre: lustre-OST0000: new disk, initializing [ 4223.316830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4223.333359] Lustre: 121768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4223.514750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.474385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4232.831170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.838431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4233.053246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4242.457833] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4248.278489] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4248.572830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4253.457733] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4253.646296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4258.787866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4258.790627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.804675] LustreError: Skipped 1 previous similar message [ 4258.807041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4258.823532] Lustre: Skipped 1 previous similar message [ 4261.558612] Lustre: server umount lustre-OST0000 complete [ 4266.464763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4266.470084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.484869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4266.493714] Lustre: Skipped 1 previous similar message [ 4271.761209] Lustre: server umount lustre-MDT0000 complete [ 4274.161226] LustreError: 119814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.203559] LustreError: 119814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4276.261223] LustreError: 120597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422080 with bad export cookie 15169743551622249254 [ 4276.286914] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4276.297815] LustreError: 120597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4276.732829] Lustre: server umount lustre-MDT0001 complete [ 4285.647309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.022235] LustreError: 123988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.110834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4291.181435] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4291.568548] LustreError: 123989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4296.676343] LustreError: 123988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4300.568484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.825549] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4315.816420] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4321.982883] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4329.845724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.613651] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4344.853651] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4351.734486] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4356.073040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4356.081086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.085205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4356.108285] Lustre: Skipped 2 previous similar messages [ 4356.113893] Lustre: Skipped 2 previous similar messages [ 4357.515966] Lustre: server umount lustre-OST0000 complete [ 4367.723715] Lustre: server umount lustre-MDT0000 complete [ 4371.388811] LustreError: 124728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422175 with bad export cookie 15169743551622250136 [ 4371.390776] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.395575] LustreError: 124728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4371.736551] Lustre: server umount lustre-MDT0001 complete [ 4386.159181] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4389.289300] Key type lgssc unregistered [ 4389.584227] LNet: 127624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.592309] LNetError: 127624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4390.632056] LNet: Removed LNI 192.168.204.106@tcp [ 4391.787221] Key type .llcrypt unregistered [ 4391.789411] Key type ._llcrypt unregistered [ 4414.393132] Key type ._llcrypt registered [ 4414.395070] Key type .llcrypt registered [ 4414.519394] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4432.243991] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4433.469284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4433.545491] alg: No test for adler32 (adler32-zlib) [ 4435.244210] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 4435.630964] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4437.391182] Key type lgssc registered [ 4438.910854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4479.546385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.614070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4481.028097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4481.093342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4481.290879] Lustre: lustre-MDT0000: new disk, initializing [ 4481.436808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.479639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4486.776638] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4497.843563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.987353] Lustre: 130778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4498.043437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4498.048947] Lustre: Skipped 1 previous similar message [ 4498.108476] Lustre: lustre-MDT0001: new disk, initializing [ 4498.215870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.253994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4498.265513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4503.415984] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4514.088562] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4521.781320] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4530.230747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4530.794987] Lustre: lustre-OST0000: new disk, initializing [ 4530.798923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4530.805450] Lustre: 132749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4530.921691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.409064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4537.427837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4537.586855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4538.206268] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4549.612842] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4554.802928] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4555.033993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4559.684680] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.867261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.427777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.434703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.462280] LustreError: Skipped 1 previous similar message [ 4563.467520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4563.495669] Lustre: Skipped 1 previous similar message [ 4568.554096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4568.560858] Lustre: Skipped 1 previous similar message [ 4573.669076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.681032] Lustre: Skipped 2 previous similar messages [ 4576.223740] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4576.483871] Lustre: server umount lustre-OST0000 complete [ 4583.905297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.911791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.923216] Lustre: Skipped 1 previous similar message [ 4587.672682] Lustre: server umount lustre-MDT0000 complete [ 4591.403724] LustreError: 130772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422395 with bad export cookie 7263184146123025735 [ 4591.408167] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4591.425296] LustreError: 130772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4591.684970] Lustre: server umount lustre-MDT0001 complete [ 4602.851494] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 00:26:44 (1786422404) [ 4610.387891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.797442] LustreError: 135163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.831400] LustreError: 135163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4610.913233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.264580] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4616.167013] LustreError: 135164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.260444] LustreError: 135163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.319709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.712260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4629.233920] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4636.842540] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4642.830782] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4645.347950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4645.348994] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.355469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4645.355477] Lustre: Skipped 1 previous similar message [ 4645.401698] Lustre: Skipped 1 previous similar message [ 4651.308646] Lustre: server umount lustre-MDT0000 complete [ 4654.560151] LustreError: 135163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4654.584507] LustreError: 135163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4655.139512] LustreError: 135143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422459 with bad export cookie 7263184146123026617 [ 4655.140969] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.147844] LustreError: 135143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4655.434565] Lustre: server umount lustre-MDT0001 complete [ 4664.079757] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 00:27:46 (1786422466) [ 4670.790363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.368077] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4691.487756] LustreError: 137581:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 4717.215396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.160148] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4733.102964] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4734.977994] Lustre: server umount lustre-OST0000 complete [ 4744.398853] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 00:29:06 (1786422546) [ 4750.569675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.906518] LustreError: 139026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.974878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.771583] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4760.572289] LustreError: 139027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.606754] LustreError: 139027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4764.336313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.782793] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4777.381742] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4782.601969] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4789.548716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4789.874124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.884241] Lustre: Skipped 1 previous similar message [ 4796.606144] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4804.276430] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4819.942576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.944678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.964852] Lustre: Skipped 2 previous similar messages [ 4819.976430] Lustre: Skipped 5 previous similar messages [ 4823.765078] Lustre: server umount lustre-MDT0000 complete [ 4825.060513] LustreError: 140814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4825.077345] LustreError: 140814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4827.853018] LustreError: 139006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422631 with bad export cookie 7263184146123027282 [ 4827.881389] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4827.890363] LustreError: 139006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4828.208619] Lustre: server umount lustre-MDT0001 complete [ 4842.792117] Lustre: server umount lustre-OST0000 complete [ 4854.138059] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 00:30:56 (1786422656) [ 4862.090203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.554145] LustreError: 142704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.733938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4868.247944] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4879.892120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.735628] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4895.903832] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4904.797398] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4913.972260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4923.150504] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4931.393410] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4933.706883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4938.409391] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4938.697152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4944.170581] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4944.370874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4948.970269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.974414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.995116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4949.018684] Lustre: Skipped 1 previous similar message [ 4949.046216] Lustre: Skipped 1 previous similar message [ 4961.247316] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4961.436260] Lustre: server umount lustre-OST0000 complete [ 4967.393779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.404285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4971.575261] Lustre: server umount lustre-MDT0000 complete [ 4974.563885] LustreError: 142701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.614188] LustreError: 142701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4975.683031] LustreError: 144550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422779 with bad export cookie 7263184146123028353 [ 4975.693588] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4975.727023] LustreError: 144550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4976.281251] Lustre: server umount lustre-MDT0001 complete [ 4987.999352] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 00:33:10 (1786422790) [ 4995.020439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5010.719659] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5015.840103] LustreError: 146868:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 5041.567418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.578163] Lustre: Skipped 2 previous similar messages [ 5049.212800] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5058.148135] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5066.371879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.064721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5073.130503] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5083.068972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.973616] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5097.542271] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5104.184595] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5110.356852] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5110.621522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5116.346263] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5116.524724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5119.458045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.469370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5119.470506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.470512] Lustre: Skipped 7 previous similar messages [ 5119.472979] Lustre: Skipped 2 previous similar messages [ 5125.471236] Lustre: server umount lustre-MDT0000 complete [ 5129.714044] LustreError: 148883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.757554] LustreError: 148883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5130.220759] LustreError: 146876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786422934 with bad export cookie 7263184146123029375 [ 5130.237567] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5130.251123] LustreError: 146876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5130.681411] Lustre: server umount lustre-MDT0001 complete [ 5145.893686] Lustre: server umount lustre-OST0000 complete [ 5155.684044] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 00:35:57 (1786422957) [ 5163.756104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.295707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5184.415824] LustreError: 151066:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 5217.283799] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5226.164025] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5234.977541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.446697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5239.662936] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5251.273043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.221522] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5264.406914] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5271.178326] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5278.938761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.214745] Lustre: lustre-OST0001: new disk, initializing [ 5279.227608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5279.239442] Lustre: 153941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5288.054146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5288.075949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5288.262629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5288.947306] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5298.650451] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5304.384803] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5304.778716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5310.547723] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5310.841872] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5314.033469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.042986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5314.059752] Lustre: Skipped 2 previous similar messages [ 5314.066146] Lustre: Skipped 5 previous similar messages [ 5319.675463] Lustre: server umount lustre-OST0000 complete [ 5324.257521] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5324.278723] LustreError: Skipped 1 previous similar message [ 5330.342535] Lustre: server umount lustre-OST0001 complete [ 5340.958303] Lustre: server umount lustre-MDT0000 complete [ 5345.823448] LustreError: 151072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423149 with bad export cookie 7263184146123030327 [ 5345.826266] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5345.836717] LustreError: 151072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5346.265398] Lustre: server umount lustre-MDT0001 complete [ 5376.269560] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 00:39:38 (1786423178) [ 5378.577572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5381.233783] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5383.484478] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 00:39:45 (1786423185) [ 5391.703812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.000958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5392.050099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5392.311289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.319911] Lustre: Skipped 6 previous similar messages [ 5399.244924] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5410.612854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.764451] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5410.820334] Lustre: 157504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5415.863989] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5424.587788] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5432.157910] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5440.416371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5440.652934] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5447.301774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5448.834900] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5457.949523] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5463.619588] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5464.046432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5469.725177] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5470.040370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5472.741690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.753722] LustreError: Skipped 1 previous similar message [ 5472.760772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.776453] Lustre: Skipped 4 previous similar messages [ 5472.784332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5472.791484] Lustre: Skipped 9 previous similar messages [ 5478.399770] LustreError: 159347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.472343] LustreError: 159347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5478.567508] Lustre: server umount lustre-OST0000 complete [ 5504.115348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.531959] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.553632] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5513.356194] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5523.071555] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5528.041544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.079349] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5531.648168] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5532.194767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5538.664865] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5538.868363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5546.017885] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90f1498ae000.ost_server_uuid 50 [ 5548.072563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90f1498ae000.ost_server_uuid in IDLE state after 0 sec [ 5548.564788] LustreError: 157510:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.6@tcp) returned error from blocking AST (req@00000000a5825aa6 x1873199485964544 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d07f6d1b/0x64cc037fb7e98891 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.6@tcp remote: 0xd3bf81c7c001260b expref: 5 pid: 157511 timeout: 5648 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5548.609909] LustreError: lustre-MDT0000: A client on nid 192.168.204.6@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5548.623362] LustreError: 157499:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.6@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d07f6d1b/0x64cc037fb7e98891 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.6@tcp remote: 0xd3bf81c7c001260b expref: 6 pid: 157511 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5559.648460] Lustre: server umount lustre-OST0000 complete [ 5564.897572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5564.905322] LustreError: Skipped 1 previous similar message [ 5570.641511] Lustre: server umount lustre-MDT0000 complete [ 5575.240754] LustreError: 157494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423379 with bad export cookie 7263184146123031510 [ 5575.245727] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.266180] LustreError: 157494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5581.924212] Lustre: server umount lustre-MDT0001 complete [ 5595.053511] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5598.721991] Key type lgssc unregistered [ 5599.290858] LNet: 163622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5599.304695] LNetError: 163622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5599.322734] LNet: Removed LNI 192.168.204.106@tcp [ 5600.196356] Key type .llcrypt unregistered [ 5600.199725] Key type ._llcrypt unregistered [ 5630.136842] Key type ._llcrypt registered [ 5630.140615] Key type .llcrypt registered [ 5630.286624] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5631.547694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5631.766778] alg: No test for adler32 (adler32-zlib) [ 5633.143670] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 5633.423218] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5635.144351] Key type lgssc registered [ 5636.587573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5648.729347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5648.750205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.445398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.606991] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5660.643655] LustreError: 165039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5660.659256] LustreError: 165039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5665.789523] LustreError: 165038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.022930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5668.321291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5674.211970] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5685.189129] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5692.491978] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.785929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.182907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5710.812743] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5720.908793] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5721.699576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5727.269878] Lustre: Failing over lustre-MDT0000 [ 5727.750778] Lustre: server umount lustre-MDT0000 complete [ 5728.447279] LustreError: 167530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.762166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5729.776626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.816553] LustreError: 165038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5731.821413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.851849] LustreError: 165038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5731.882903] Lustre: Skipped 1 previous similar message [ 5736.928851] LustreError: 165767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.948293] LustreError: 165767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5741.115488] LustreError: 165020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.6@tcp arrived at 1786423545 with bad export cookie 8281215470959837152 [ 5747.167233] Lustre: 164347:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423535/real 1786423535] req@000000002a5baa6a x1873200741906688/t0(0) o400->MGC192.168.204.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786423551 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5747.175623] LustreError: 165035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.215647] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5747.246216] LustreError: 165035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5752.960259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.937183] LustreError: 164346:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b08dc4fe x1873200741912064/t0(0) o250->MGC192.168.204.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5758.481881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.607640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5763.586397] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5763.606324] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5766.109389] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5768.943980] Lustre: lustre-MDT0000: Denying connection for new client 973024b2-3e96-475a-a1ed-1b856827d752 (at 192.168.204.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 5773.964642] Lustre: lustre-MDT0000: Denying connection for new client 973024b2-3e96-475a-a1ed-1b856827d752 (at 192.168.204.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5778.919599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.948172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5784.104171] Lustre: server umount lustre-OST0000 complete [ 5789.096550] LustreError: 168774:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5789.112547] Lustre: 168123:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5789.121757] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5789.145696] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5789.155392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.155971] LustreError: 168123:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000019a4abb x1873200741927680/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5789.159893] Lustre: Skipped 1 previous similar message [ 5789.167099] LustreError: 168123:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5789.169149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5789.174717] LustreError: 168123:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5789.176950] Lustre: Skipped 1 previous similar message [ 5789.183970] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 5789.385314] Lustre: server umount lustre-MDT0000 complete [ 5794.196752] LustreError: 165019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423598 with bad export cookie 8281215470959837481 [ 5794.214604] LustreError: 165019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5794.220561] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.222424] LustreError: 165038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.222434] LustreError: 165038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5800.739464] Lustre: server umount lustre-MDT0001 complete [ 5818.243386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5822.654362] Key type lgssc unregistered [ 5822.928914] LNet: 169533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5822.936687] LNetError: 169533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5823.976171] LNet: Removed LNI 192.168.204.106@tcp [ 5825.628641] Key type .llcrypt unregistered [ 5825.633417] Key type ._llcrypt unregistered [ 5864.069654] Key type ._llcrypt registered [ 5864.074562] Key type .llcrypt registered [ 5864.268726] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5866.801568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5866.940043] alg: No test for adler32 (adler32-zlib) [ 5868.610330] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 5868.847146] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5870.631419] Key type lgssc registered [ 5872.420609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5882.576924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5882.607766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.255689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5891.546475] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5894.630415] LustreError: 170947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.660566] LustreError: 170947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5899.750600] LustreError: 170948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.331560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5902.738231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5909.571460] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5920.914797] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.286481] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5937.935442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5938.351244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5947.063247] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5954.744527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5958.210470] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5965.284523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.293226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5965.295107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5965.301444] Lustre: Skipped 1 previous similar message [ 5970.431773] Lustre: server umount lustre-OST0000 complete [ 5975.526617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5975.548724] Lustre: Skipped 1 previous similar message [ 5975.554342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5975.571361] Lustre: Skipped 2 previous similar messages [ 5980.644068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5980.658678] Lustre: Skipped 1 previous similar message [ 5985.765955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5985.770833] Lustre: Skipped 1 previous similar message [ 5989.343160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5989.571969] Lustre: server umount lustre-MDT0000 complete [ 5990.885599] LustreError: 170948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5990.904915] LustreError: 170948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5995.221683] LustreError: 170927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423799 with bad export cookie 17590852346806540925 [ 5995.233230] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5995.241330] LustreError: 170927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5995.818800] Lustre: server umount lustre-MDT0001 complete [ 6015.126328] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6019.833245] Key type lgssc unregistered [ 6020.334640] LNet: 174504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6020.346480] LNetError: 174504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6020.379886] LNet: Removed LNI 192.168.204.106@tcp [ 6022.027155] Key type .llcrypt unregistered [ 6022.034459] Key type ._llcrypt unregistered [ 6065.157389] Key type ._llcrypt registered [ 6065.167546] Key type .llcrypt registered [ 6065.411170] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6067.066270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6067.227227] alg: No test for adler32 (adler32-zlib) [ 6068.543860] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6068.957908] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6070.639290] Key type lgssc registered [ 6072.457971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6082.385496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6082.418803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.099927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6089.710612] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6094.312186] LustreError: 176208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.369199] LustreError: 176208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6099.425199] LustreError: 176209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.884434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6101.178783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6106.725751] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6114.996752] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6122.017730] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6131.332796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6131.561027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6138.867705] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6147.919654] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6154.179591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6161.615923] Lustre: server umount lustre-OST0000 complete [ 6168.035974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6168.047066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.066438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6169.570303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6169.574408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.692763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.707096] Lustre: Skipped 2 previous similar messages [ 6178.783275] Lustre: 175519:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423966/real 1786423966] req@00000000a31cf3b1 x1873201198036096/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786423982 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6178.808703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.825287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6178.833413] Lustre: Skipped 1 previous similar message [ 6180.319253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6180.682569] Lustre: server umount lustre-MDT0000 complete [ 6180.895382] Lustre: 175519:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786423968/real 1786423968] req@00000000dde9b2fc x1873201198037120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786423984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6183.907794] LustreError: 176205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.973806] LustreError: 176205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6185.580230] LustreError: 176189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786423989 with bad export cookie 5803168185025017117 [ 6185.583975] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.612203] LustreError: 176189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6186.202825] Lustre: server umount lustre-MDT0001 complete [ 6201.336827] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6204.577907] Key type lgssc unregistered [ 6204.987706] LNet: 179749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6205.006892] LNetError: 179749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6205.025723] LNet: Removed LNI 192.168.204.106@tcp [ 6206.172843] Key type .llcrypt unregistered [ 6206.174924] Key type ._llcrypt unregistered [ 6232.143747] Key type ._llcrypt registered [ 6232.148165] Key type .llcrypt registered [ 6232.383801] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6233.793578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6233.848089] alg: No test for adler32 (adler32-zlib) [ 6235.034390] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6235.353875] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6236.999189] Key type lgssc registered [ 6238.140781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6250.190110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6250.204441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6251.586576] Lustre: *** cfs_fail_loc=135, val=0*** [ 6251.669652] LustreError: 181273:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6251.677933] LustreError: 181273:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -2 [ 6251.696682] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6251.696682] [ 6251.709540] LustreError: MGC192.168.204.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6251.721917] LustreError: 181235:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6251.731806] LustreError: 181235:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6251.743629] LustreError: 181235:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6251.750636] LustreError: 181235:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6251.826501] Lustre: server umount lustre-MDT0000 complete [ 6251.833780] LustreError: 181235:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6259.424242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6259.820094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6266.430076] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6269.412572] LustreError: 181646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.437493] LustreError: 181646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6274.545723] LustreError: 181647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.807116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.162594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6283.395321] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6293.342758] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6300.347713] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6303.588205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.610394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6303.615249] Lustre: Skipped 1 previous similar message [ 6308.835903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.847302] Lustre: Skipped 1 previous similar message [ 6308.955331] Lustre: server umount lustre-MDT0000 complete [ 6313.170937] LustreError: 181627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424117 with bad export cookie 10741621358718708972 [ 6313.174584] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6313.179167] LustreError: 181627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6313.447621] Lustre: server umount lustre-MDT0001 complete [ 6327.355327] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6330.879886] Key type lgssc unregistered [ 6331.272510] LNet: 183971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6331.285752] LNetError: 183971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6332.326701] LNet: Removed LNI 192.168.204.106@tcp [ 6334.007716] Key type .llcrypt unregistered [ 6334.015578] Key type ._llcrypt unregistered [ 6361.983718] Key type ._llcrypt registered [ 6361.985435] Key type .llcrypt registered [ 6362.136986] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6362.480406] Key type .llcrypt unregistered [ 6362.485959] Key type ._llcrypt unregistered [ 6383.523891] Key type ._llcrypt registered [ 6383.527128] Key type .llcrypt registered [ 6383.696509] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6385.080519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6385.198209] alg: No test for adler32 (adler32-zlib) [ 6386.276938] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6386.606430] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6388.295175] Key type lgssc registered [ 6389.715765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6397.793427] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6397.805326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6413.343503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6418.464221] LustreError: 186111:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 6445.087833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6452.443344] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6461.882311] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6470.542687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.943032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.018341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6476.076423] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6476.267983] LustreError: 186137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.297212] LustreError: 186137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6479.346105] LustreError: 186136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.400582] LustreError: 186136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6484.453089] LustreError: 187577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6484.474523] LustreError: 187577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6484.999343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.342586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6490.690440] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6498.697684] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6504.669619] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6507.619773] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6521.321179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.325682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.356855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6526.437186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6526.440147] Lustre: Skipped 2 previous similar messages [ 6527.479392] Lustre: server umount lustre-OST0000 complete [ 6531.553897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6531.561137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.565744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6531.573157] Lustre: Skipped 2 previous similar messages [ 6531.582742] Lustre: Skipped 1 previous similar message [ 6536.676338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6536.692244] Lustre: Skipped 1 previous similar message [ 6537.936205] Lustre: server umount lustre-MDT0000 complete [ 6541.803695] LustreError: 187577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6541.871288] LustreError: 187577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6542.877673] LustreError: 188774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424346 with bad export cookie 7192478865442581919 [ 6542.891807] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.900535] LustreError: 188774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6543.338949] Lustre: server umount lustre-MDT0001 complete [ 6556.391650] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6559.877679] Key type lgssc unregistered [ 6560.205906] LNet: 190129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6560.217206] LNetError: 190129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6560.228677] LNet: Removed LNI 192.168.204.106@tcp [ 6561.144689] Key type .llcrypt unregistered [ 6561.147576] Key type ._llcrypt unregistered [ 6587.797579] Key type ._llcrypt registered [ 6587.802172] Key type .llcrypt registered [ 6588.015508] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6590.245291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6590.372142] alg: No test for adler32 (adler32-zlib) [ 6591.589293] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6591.835452] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6593.511176] Key type lgssc registered [ 6595.370674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6606.136801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6606.172854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.055297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6614.672507] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6618.595609] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.612362] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6623.716928] LustreError: 191545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.654148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.969089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6632.508769] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6640.794918] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6647.946279] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6655.138240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.478899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.176519] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6672.142935] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.058812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6682.341348] Lustre: Failing over lustre-MDT0000 [ 6682.537644] Lustre: server umount lustre-MDT0000 complete [ 6684.135860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.150982] Lustre: Skipped 2 previous similar messages [ 6684.168151] LustreError: 191545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6684.201045] LustreError: 191545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6689.248304] LustreError: 191539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.266950] LustreError: 191539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6694.372486] LustreError: 192467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.393316] LustreError: 192467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6700.512909] Lustre: 190854:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786424488/real 1786424488] req@000000006cb6ac85 x1873201746440192/t0(0) o400->MGC192.168.204.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786424504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6700.566963] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6702.323574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.775501] LustreError: 191539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.790995] LustreError: 191539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6711.202160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6711.280283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6715.023337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6716.440437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6716.495745] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6716.582941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6719.640883] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6725.104174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6735.568450] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6742.507179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6742.511624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.533518] LustreError: Skipped 1 previous similar message [ 6742.548424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6744.419732] Lustre: server umount lustre-OST0000 complete [ 6752.227295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.229096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6752.234507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6752.234515] Lustre: Skipped 1 previous similar message [ 6752.245301] Lustre: Skipped 2 previous similar messages [ 6756.266222] Lustre: server umount lustre-MDT0000 complete [ 6757.349394] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.414874] LustreError: 191544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6761.196186] LustreError: 191526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424565 with bad export cookie 18097695425255328144 [ 6761.201013] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6761.213740] LustreError: 191526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6761.568873] Lustre: server umount lustre-MDT0001 complete [ 6776.391386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6780.187558] Key type lgssc unregistered [ 6780.509611] LNet: 196496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.524898] LNetError: 196496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6780.549960] LNet: Removed LNI 192.168.204.106@tcp [ 6781.889959] Key type .llcrypt unregistered [ 6781.893204] Key type ._llcrypt unregistered [ 6810.759989] Key type ._llcrypt registered [ 6810.764323] Key type .llcrypt registered [ 6810.882557] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6811.923968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6812.163167] alg: No test for adler32 (adler32-zlib) [ 6813.307446] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6813.600072] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6815.439198] Key type lgssc registered [ 6817.161980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6826.205156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6826.219755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.789058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6832.819848] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6837.228751] LustreError: 197911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6837.250739] LustreError: 197911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6842.072439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.414657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6847.619701] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6855.499374] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.740382] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.313672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.697207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.783683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6877.560872] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 6885.979620] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6890.777576] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6899.182389] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6916.960766] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6929.380147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6929.390168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6929.406966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.479202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.489313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6933.526242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.740284] Lustre: server umount lustre-OST0000 complete [ 6939.616172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6939.625160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.644442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6939.652391] Lustre: Skipped 1 previous similar message [ 6943.713423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6943.718591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6945.592385] Lustre: server umount lustre-MDT0000 complete [ 6948.836535] LustreError: 197905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6948.871395] LustreError: 197905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6950.136554] LustreError: 197890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424754 with bad export cookie 4779414258949242413 [ 6950.143863] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6950.165596] LustreError: 197890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6950.695420] Lustre: server umount lustre-MDT0001 complete [ 6963.306785] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 6966.520692] Key type lgssc unregistered [ 6967.056932] LNet: 201790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6967.069426] LNetError: 201790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6968.103382] LNet: Removed LNI 192.168.204.106@tcp [ 6969.336459] Key type .llcrypt unregistered [ 6969.339539] Key type ._llcrypt unregistered [ 6994.079857] Key type ._llcrypt registered [ 6994.083232] Key type .llcrypt registered [ 6994.187398] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 6995.341432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6995.383615] alg: No test for adler32 (adler32-zlib) [ 6996.540213] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 6996.845435] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 6998.599793] Key type lgssc registered [ 7000.365607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7008.608733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7008.623474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.008377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.228196] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7019.556482] LustreError: 203056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7019.578944] LustreError: 203056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7023.961311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7024.305658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7031.367807] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7039.645120] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.191856] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.048959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.491332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.892348] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7071.328360] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.105566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 7089.167779] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7099.107490] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7108.106763] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7118.468780] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7130.596029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.599069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7130.633533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7132.988461] Lustre: server umount lustre-OST0000 complete [ 7140.837505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7140.862619] Lustre: Skipped 2 previous similar messages [ 7140.872208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7140.891852] Lustre: Skipped 2 previous similar messages [ 7143.854798] Lustre: server umount lustre-MDT0000 complete [ 7145.978475] LustreError: 203053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.021270] LustreError: 203053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7148.083045] LustreError: 203038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786424952 with bad export cookie 5972124805576223588 [ 7148.091961] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.097776] LustreError: 203038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7148.492259] Lustre: server umount lustre-MDT0001 complete [ 7160.297734] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7163.950606] Key type lgssc unregistered [ 7164.194795] LNet: 207957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7164.201706] LNetError: 207957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7164.223862] LNet: Removed LNI 192.168.204.106@tcp [ 7165.207536] Key type .llcrypt unregistered [ 7165.209681] Key type ._llcrypt unregistered [ 7193.522214] Key type ._llcrypt registered [ 7193.528249] Key type .llcrypt registered [ 7193.729212] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7195.127200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7195.178968] alg: No test for adler32 (adler32-zlib) [ 7196.312954] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 7196.644106] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7198.447235] Key type lgssc registered [ 7199.575839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7208.573819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7208.586524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7210.079559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.033869] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7220.195183] LustreError: 209371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7220.209502] LustreError: 209371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7225.317073] LustreError: 209370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.283193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.690229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7231.465188] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7239.899896] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.098468] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7255.118046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.422360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.475091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7262.727272] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7271.300175] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7287.266748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7287.269382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7287.271173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7287.282238] Lustre: Skipped 1 previous similar message [ 7287.315971] Lustre: Skipped 1 previous similar message [ 7292.390265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7292.402643] Lustre: Skipped 1 previous similar message [ 7292.662336] Lustre: server umount lustre-OST0000 complete [ 7297.506191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.516467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7297.520633] Lustre: Skipped 1 previous similar message [ 7302.635784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7302.644208] Lustre: Skipped 1 previous similar message [ 7302.924877] Lustre: server umount lustre-MDT0000 complete [ 7306.430603] LustreError: 209352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425110 with bad export cookie 6567320804699004644 [ 7306.438382] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.445228] LustreError: 209352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7306.742268] Lustre: server umount lustre-MDT0001 complete [ 7318.794935] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7321.576944] Key type lgssc unregistered [ 7321.776331] LNet: 213060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7321.791299] LNetError: 213060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7321.800524] LNet: Removed LNI 192.168.204.106@tcp [ 7322.909227] Key type .llcrypt unregistered [ 7322.911819] Key type ._llcrypt unregistered [ 7341.568284] Key type ._llcrypt registered [ 7341.570034] Key type .llcrypt registered [ 7341.688924] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7343.173386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7343.183639] alg: No test for adler32 (adler32-zlib) [ 7344.284403] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 7344.492174] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7346.191170] Key type lgssc registered [ 7347.662397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7355.833905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7355.852478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.212260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7361.864915] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7367.651201] LustreError: 214325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.681289] LustreError: 214325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7370.821303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7371.121230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7376.460540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7385.360313] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7392.035185] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7400.084082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7400.485715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7402.571902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7408.489891] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7417.456473] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7436.259774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7436.266249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.268217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.283963] Lustre: Skipped 1 previous similar message [ 7440.093022] Lustre: server umount lustre-OST0000 complete [ 7446.497716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.523581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7446.539410] Lustre: Skipped 2 previous similar messages [ 7449.919678] Lustre: server umount lustre-MDT0000 complete [ 7451.620275] LustreError: 215810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7451.649934] LustreError: 215810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7454.119534] LustreError: 214305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425258 with bad export cookie 4464479720710039652 [ 7454.128845] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7454.136659] LustreError: 214305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7454.421450] Lustre: server umount lustre-MDT0001 complete [ 7467.899345] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7470.840407] Key type lgssc unregistered [ 7471.249819] LNet: 218011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7471.259832] LNetError: 218011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7472.295557] LNet: Removed LNI 192.168.204.106@tcp [ 7473.304401] Key type .llcrypt unregistered [ 7473.313088] Key type ._llcrypt unregistered [ 7496.220188] Key type ._llcrypt registered [ 7496.223399] Key type .llcrypt registered [ 7496.344949] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7497.462974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7497.616392] alg: No test for adler32 (adler32-zlib) [ 7499.046659] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 7499.293904] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7501.055167] Key type lgssc registered [ 7502.486161] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7511.637530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7511.648146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.062298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7519.127907] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7522.279257] LustreError: 219281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.316727] LustreError: 219281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7527.401629] LustreError: 219280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.866123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.152718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7536.560893] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7544.774490] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7552.596744] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7561.758435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.392020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.495424] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7579.349252] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7582.928314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7595.282876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7595.333329] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7595.576671] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7603.665385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7612.142991] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7613.836626] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7625.656636] Lustre: Permanently deactivating lustre-OST0001 [ 7625.691911] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7632.491324] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7640.671298] Lustre: Permanently reactivating lustre-OST0001 [ 7640.692865] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7640.697468] Lustre: Skipped 2 previous similar messages [ 7649.860199] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.870435] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7649.883356] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7649.910482] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7649.981518] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7653.401774] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.406573] LustreError: 222792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.420207] Lustre: Skipped 2 previous similar messages [ 7653.496725] Lustre: server umount lustre-OST0001 complete [ 7657.953819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7657.962497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.988402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7658.471614] LustreError: 221106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.491925] LustreError: 221106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7663.591243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7663.592108] LustreError: 221106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.596086] Lustre: Skipped 3 previous similar messages [ 7663.611927] LustreError: 221106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7664.079859] Lustre: server umount lustre-OST0000 complete [ 7673.827853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.833586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7673.846475] Lustre: Skipped 1 previous similar message [ 7673.852358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7673.879938] Lustre: Skipped 1 previous similar message [ 7675.816267] Lustre: server umount lustre-MDT0000 complete [ 7678.949620] LustreError: 219280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7678.994585] LustreError: 219280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7679.423369] LustreError: 222123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425483 with bad export cookie 11389983282007455343 [ 7679.424452] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.435392] LustreError: 222123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7679.783175] Lustre: server umount lustre-MDT0001 complete [ 7694.160188] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 7697.201598] Key type lgssc unregistered [ 7697.541969] LNet: 224335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.552240] LNetError: 224335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.572382] LNet: Removed LNI 192.168.204.106@tcp [ 7698.483612] Key type .llcrypt unregistered [ 7698.486931] Key type ._llcrypt unregistered [ 7741.295942] Key type ._llcrypt registered [ 7741.299353] Key type .llcrypt registered [ 7741.480645] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 7743.008365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7743.172252] alg: No test for adler32 (adler32-zlib) [ 7744.555763] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 7744.878359] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 7746.591506] Key type lgssc registered [ 7747.875190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7756.962423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7756.981364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.183734] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7758.202629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7758.388936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7763.624685] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7773.778836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.846990] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7773.875308] Lustre: 226539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7774.228552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7780.414319] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7789.242511] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7795.510327] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7804.143816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7804.335956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7804.548936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.769537] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 7814.480079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7820.175768] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7823.884440] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7831.887924] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7840.014860] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7845.952313] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7856.715340] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7856.730387] Lustre: Skipped 1 previous similar message [ 7876.995784] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7877.001129] Lustre: Skipped 2 previous similar messages [ 7914.665113] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7914.681496] Lustre: Skipped 4 previous similar messages [ 7983.594604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.598135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.607212] LustreError: Skipped 1 previous similar message [ 7983.610166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.666058] Lustre: Skipped 1 previous similar message [ 7988.706552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7988.718407] Lustre: Skipped 2 previous similar messages [ 7989.595365] Lustre: server umount lustre-OST0000 complete [ 7994.342778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7994.352706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.377816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7998.945835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7998.947489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.384550] Lustre: server umount lustre-MDT0000 complete [ 8004.616369] LustreError: 226532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425808 with bad export cookie 7240532896190653150 [ 8004.631768] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8004.938656] Lustre: server umount lustre-MDT0001 complete [ 8017.622893] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8020.608471] Key type lgssc unregistered [ 8020.980717] LNet: 231333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8020.995623] LNetError: 231333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8021.014370] LNet: Removed LNI 192.168.204.106@tcp [ 8022.087314] Key type .llcrypt unregistered [ 8022.090634] Key type ._llcrypt unregistered [ 8048.458859] Key type ._llcrypt registered [ 8048.461362] Key type .llcrypt registered [ 8048.558291] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8050.038694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8050.163403] alg: No test for adler32 (adler32-zlib) [ 8051.443859] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 8051.704482] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8053.415637] Key type lgssc registered [ 8055.290872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.139401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8065.166898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.795507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.128951] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8076.323869] LustreError: 232746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.365400] LustreError: 232746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8081.037408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8081.484903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8085.630729] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8093.244730] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8099.762345] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8109.049946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8109.417978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8111.528498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 8117.015089] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 8125.614746] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8147.428461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8147.432181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8147.443982] LustreError: Skipped 1 previous similar message [ 8147.447275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8147.478487] Lustre: Skipped 1 previous similar message [ 8147.491318] Lustre: Skipped 1 previous similar message [ 8149.368479] Lustre: server umount lustre-OST0000 complete [ 8157.664853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.672211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8157.684547] Lustre: Skipped 1 previous similar message [ 8160.504482] Lustre: server umount lustre-MDT0000 complete [ 8162.784742] LustreError: 232747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8162.806349] LustreError: 232747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8164.572581] LustreError: 232727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786425968 with bad export cookie 2922018670046738301 [ 8164.580945] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8164.593353] LustreError: 232727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8164.856822] Lustre: server umount lustre-MDT0001 complete [ 8178.198918] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8181.411657] Key type lgssc unregistered [ 8181.941729] LNet: 236445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8181.947103] LNetError: 236445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8181.964692] LNet: Removed LNI 192.168.204.106@tcp [ 8183.215129] Key type .llcrypt unregistered [ 8183.217621] Key type ._llcrypt unregistered [ 8211.501290] Key type ._llcrypt registered [ 8211.503629] Key type .llcrypt registered [ 8211.644346] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8213.597454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8213.613756] alg: No test for adler32 (adler32-zlib) [ 8214.988825] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 8215.228617] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8216.911175] Key type lgssc registered [ 8218.635392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8492.038496] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8496.892548] Key type lgssc unregistered [ 8497.426823] LNet: 238473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8497.443980] LNetError: 238473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8497.460243] LNet: Removed LNI 192.168.204.106@tcp [ 8499.086131] Key type .llcrypt unregistered [ 8499.100663] Key type ._llcrypt unregistered [ 8527.945554] Key type ._llcrypt registered [ 8527.948701] Key type .llcrypt registered [ 8528.076641] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 8549.059556] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 8550.394141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8550.506626] alg: No test for adler32 (adler32-zlib) [ 8552.014246] Lustre: Lustre: Build Version: 2.17.56_52_gb6f4e2d [ 8552.321982] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 8554.031805] Key type lgssc registered [ 8555.146970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8590.801799] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8207 sec ========== 01:33:13 (1786426393) [ 8592.856294] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:33:15 (1786426395) === [ 8597.294280] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:33:19 (1786426399) === [ 8642.026839] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 8646.142898] Key type lgssc unregistered [ 8646.501806] LNet: 243722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8646.515941] LNetError: 243722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8646.530838] LNet: Removed LNI 192.168.204.106@tcp [ 8647.310356] Key type .llcrypt unregistered [ 8647.311711] Key type ._llcrypt unregistered