[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489199823 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2544MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002363] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006814] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009038] Yama: becoming mindful. [ 0.010025] SELinux: Initializing. [ 0.011062] *** VALIDATE selinux *** [ 0.019238] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024173] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025141] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026078] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027079] *** VALIDATE tmpfs *** [ 0.028444] *** VALIDATE proc *** [ 0.030022] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034016] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037019] Speculative Store Bypass: Vulnerable [ 0.040631] debug: unmapping init [mem 0xffffffffafa59000-0xffffffffafa60fff] [ 0.043251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044679] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045017] ... version: 2 [ 0.046007] ... bit width: 48 [ 0.047006] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049007] ... max period: 00007fffffffffff [ 0.050007] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052292] rcu: Hierarchical SRCU implementation. [ 0.054330] smp: Bringing up secondary CPUs ... [ 0.055513] x86: Booting SMP configuration: [ 0.056014] .... node #0, CPUs: #1 #2 #3 [ 0.066769] smp: Brought up 1 node, 4 CPUs [ 0.068011] smpboot: Max logical packages: 1 [ 0.069009] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.132018] node 0 deferred pages initialised in 61ms [ 0.137144] devtmpfs: initialized [ 0.138217] x86/mm: Memory block size: 128MB [ 0.141063] gcov: version magic: 0x41383552 [ 0.144148] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146202] pinctrl core: initialized pinctrl subsystem [ 0.147140] [ 0.147894] ************************************************************* [ 0.148007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149008] ** ** [ 0.150008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151008] ** ** [ 0.152009] ** This means that this kernel is built to expose internal ** [ 0.153008] ** IOMMU data structures, which may compromise security on ** [ 0.154009] ** your system. ** [ 0.155012] ** ** [ 0.156011] ** If you see this message and you are not debugging the ** [ 0.157013] ** kernel, report this immediately to your vendor! ** [ 0.158018] ** ** [ 0.159017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160007] ************************************************************* [ 0.161927] NET: Registered protocol family 16 [ 0.162528] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165723] cpuidle: using governor menu [ 0.168055] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.169367] PCI: Using configuration type 1 for base access [ 0.170109] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181397] cryptd: max_cpu_qlen set to 1000 [ 0.183551] ACPI: Added _OSI(Module Device) [ 0.184000] ACPI: Added _OSI(Processor Device) [ 0.184010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190718] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198231] ACPI: Interpreter enabled [ 0.199045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200009] ACPI: Using IOAPIC for interrupt routing [ 0.201082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202452] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213216] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218713] acpiphp: Slot [2] registered [ 0.219089] acpiphp: Slot [5] registered [ 0.220085] acpiphp: Slot [6] registered [ 0.221096] acpiphp: Slot [7] registered [ 0.222099] acpiphp: Slot [8] registered [ 0.223102] acpiphp: Slot [9] registered [ 0.224071] acpiphp: Slot [10] registered [ 0.225090] acpiphp: Slot [3] registered [ 0.226079] acpiphp: Slot [4] registered [ 0.227063] acpiphp: Slot [11] registered [ 0.228076] acpiphp: Slot [12] registered [ 0.229101] acpiphp: Slot [13] registered [ 0.230195] acpiphp: Slot [14] registered [ 0.231073] acpiphp: Slot [15] registered [ 0.232073] acpiphp: Slot [16] registered [ 0.233069] acpiphp: Slot [17] registered [ 0.234091] acpiphp: Slot [18] registered [ 0.235082] acpiphp: Slot [19] registered [ 0.236108] acpiphp: Slot [20] registered [ 0.237102] acpiphp: Slot [21] registered [ 0.239103] acpiphp: Slot [22] registered [ 0.240234] acpiphp: Slot [23] registered [ 0.241079] acpiphp: Slot [24] registered [ 0.242077] acpiphp: Slot [25] registered [ 0.243088] acpiphp: Slot [26] registered [ 0.244095] acpiphp: Slot [27] registered [ 0.245081] acpiphp: Slot [28] registered [ 0.246078] acpiphp: Slot [29] registered [ 0.247138] acpiphp: Slot [30] registered [ 0.248072] acpiphp: Slot [31] registered [ 0.249054] PCI host bridge to bus 0000:00 [ 0.250014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.253059] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.254019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.255019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.258674] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.264106] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290062] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.299030] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.308017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.319019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323167] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331684] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.338038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344130] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400035] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.488068] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.519015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.541013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.565111] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.579015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.597014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.630024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.650272] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.668012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.701012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.714000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.729013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.742013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.766012] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.778598] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.785008] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.790010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.810012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.822947] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.824325] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.826341] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.830291] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.831000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.836098] iommu: Default domain type: Passthrough [ 0.838382] SCSI subsystem initialized [ 0.840154] ACPI: bus type USB registered [ 0.843140] usbcore: registered new interface driver usbfs [ 0.846072] usbcore: registered new interface driver hub [ 0.848065] usbcore: registered new device driver usb [ 0.850201] pps_core: LinuxPPS API ver. 1 registered [ 0.853008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.855035] PTP clock support registered [ 0.857035] EDAC MC: Ver: 3.0.0 [ 0.859053] PCI: Using ACPI for IRQ routing [ 0.860941] NetLabel: Initializing [ 0.863008] NetLabel: domain hash size = 128 [ 0.864007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.866076] NetLabel: unlabeled traffic allowed by default [ 0.869357] vgaarb: loaded [ 0.870255] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.872010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.881496] clocksource: Switched to clocksource kvm-clock [ 1.076785] VFS: Disk quotas dquot_6.6.0 [ 1.078507] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.081304] *** VALIDATE ramfs *** [ 1.082659] *** VALIDATE hugetlbfs *** [ 1.084051] pnp: PnP ACPI init [ 1.086880] pnp: PnP ACPI: found 6 devices [ 1.103279] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.106651] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.108964] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.111499] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.114029] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.116369] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.119151] NET: Registered protocol family 2 [ 1.121542] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.126424] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.130483] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.135825] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.139447] TCP: Hash tables configured (established 65536 bind 65536) [ 1.142581] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.145638] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.148613] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.151599] NET: Registered protocol family 1 [ 1.154539] RPC: Registered named UNIX socket transport module. [ 1.155764] RPC: Registered udp transport module. [ 1.156756] RPC: Registered tcp transport module. [ 1.157773] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.159386] NET: Registered protocol family 44 [ 1.161130] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.163418] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.165616] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.168183] PCI: CLS 0 bytes, default 64 [ 1.170130] Unpacking initramfs... [ 4.776633] debug: unmapping init [mem 0xffff962ebcc54000-0xffff962ebffbffff] [ 4.817483] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.820767] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 4.831438] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 5.663651] Initialise system trusted keyrings [ 5.667193] Key type blacklist registered [ 5.669709] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.682680] zbud: loaded [ 5.688030] *** VALIDATE nfs *** [ 5.691056] *** VALIDATE nfs4 *** [ 5.693095] pstore: using deflate compression [ 5.701063] Platform Keyring initialized [ 5.891560] NET: Registered protocol family 38 [ 5.894618] Key type asymmetric registered [ 5.898093] Asymmetric key parser 'x509' registered [ 5.901291] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.904860] io scheduler mq-deadline registered [ 5.906811] io scheduler kyber registered [ 5.908866] io scheduler bfq registered [ 5.910963] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.915110] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.919246] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.932937] ACPI: Power Button [PWRF] [ 5.949379] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.972572] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.015309] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.038546] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.088036] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.135671] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.173029] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.193199] Non-volatile memory driver v1.3 [ 6.197133] Linux agpgart interface v0.103 [ 6.269852] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 6.282323] vda: detected capacity change from 0 to 69627904 [ 6.324800] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.336129] vdb: detected capacity change from 0 to 1073741824 [ 6.385536] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.395736] vdc: detected capacity change from 0 to 2621440000 [ 6.440750] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.448990] vdd: detected capacity change from 0 to 2621440000 [ 6.493487] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.500841] vde: detected capacity change from 0 to 4294967296 [ 6.550133] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.556629] vdf: detected capacity change from 0 to 4294967296 [ 6.583673] libphy: Fixed MDIO Bus: probed [ 6.603851] usbcore: registered new interface driver usbserial_generic [ 6.614834] usbserial: USB Serial support registered for generic [ 6.617782] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.634955] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.641862] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.654977] mousedev: PS/2 mouse device common for all mice [ 6.665505] rtc_cmos 00:05: RTC can wake from S4 [ 6.669652] rtc_cmos 00:05: registered as rtc0 [ 6.675242] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.677654] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.684964] intel_pstate: CPU model not supported [ 6.699519] hid: raw HID events driver (C) Jiri Kosina [ 6.710035] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.712884] usbcore: registered new interface driver usbhid [ 6.723951] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.732462] usbhid: USB HID core driver [ 6.734432] drop_monitor: Initializing network drop monitor service [ 6.751554] Initializing XFRM netlink socket [ 6.756840] NET: Registered protocol family 10 [ 6.766651] Segment Routing with IPv6 [ 6.771064] NET: Registered protocol family 17 [ 6.777841] mpls_gso: MPLS GSO support [ 6.799521] RAS: Correctable Errors collector initialized. [ 6.806287] AVX version of gcm_enc/dec engaged. [ 6.812088] AES CTR mode by8 optimization enabled [ 7.027996] sched_clock: Marking stable (7027976667, 0)->(8181004851, -1153028184) [ 7.035836] registered taskstats version 1 [ 7.046350] Loading compiled-in X.509 certificates [ 7.050613] zswap: loaded using pool lzo/zbud [ 7.165779] Key type big_key registered [ 7.198695] Key type encrypted registered [ 7.200390] ima: No TPM chip found, activating TPM-bypass! [ 7.202420] ima: Allocated hash algorithm: sha1 [ 7.203938] ima: No architecture policies found [ 7.205561] evm: Initialising EVM extended attributes: [ 7.207563] evm: security.selinux [ 7.208802] evm: security.ima [ 7.210091] evm: security.capability [ 7.211653] evm: HMAC attrs: 0x1 [ 7.214310] rtc_cmos 00:05: setting system clock to 2026-06-04 00:33:34 UTC (1780533214) [ 7.221618] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 7.225071] debug: unmapping init [mem 0xffffffffaf782000-0xffffffffafa58fff] [ 7.233234] Write protecting the kernel read-only data: 28672k [ 7.268675] debug: unmapping init [mem 0xffffffffade03000-0xffffffffadffffff] [ 7.275353] debug: unmapping init [mem 0xffffffffae714000-0xffffffffae7fffff] [ 7.368073] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.376724] systemd[1]: Detected virtualization kvm. [ 7.380497] systemd[1]: Detected architecture x86-64. [ 7.385364] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.472128] systemd[1]: No hostname configured. [ 7.473894] systemd[1]: Set hostname to . [ 7.480317] random: systemd: uninitialized urandom read (16 bytes read) [ 7.484843] systemd[1]: Initializing machine ID from random generator. [ 7.605630] random: ln: uninitialized urandom read (6 bytes read) [ 7.818392] random: systemd: uninitialized urandom read (16 bytes read) [ 7.825591] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 7.833360] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.842197] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.156906] device-mapper: uevent: version 1.0.3 [ 9.159512] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 12.365458] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 12.794832] random: fast init done Starting dracut initqueue hook... [ 12.983541] scsi host0: ata_piix [ 13.308746] scsi host1: ata_piix [ 13.322299] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.336639] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.725445] random: crng init done [ 18.727053] random: 7 urandom warning(s) missed due to ratelimiting [ 19.440069] dracut-initqueue[570]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.852907] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.903699] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.518097] SELinux: Disabled at runtime. [ 24.654622] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.667560] systemd[1]: Detected virtualization kvm. [ 24.671423] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.948043] hrtimer: interrupt took 8034773 ns [ 27.519497] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.526568] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.536490] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.546459] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.552811] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.567176] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.578798] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ 27.737375] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.847150] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.713446] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.718718] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.976858] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.026703] EDAC sbridge: Ver: 1.1.2 [ 33.179312] Key type dns_resolver registered [ 33.619622] NFS: Registering the id_resolver key type [ 33.621700] Key type id_resolver registered [ 33.622959] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg402-server login: [ 58.040655] libcfs: loading out-of-tree module taints kernel. [ 58.058652] Key type ._llcrypt registered [ 58.060371] Key type .llcrypt registered [ 58.102900] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 65.543070] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 66.068516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.074481] alg: No test for adler32 (adler32-zlib) [ 67.067749] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 67.351316] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 68.960144] Key type lgssc registered [ 69.578775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.720745] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 89.727618] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 94.086588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 94.093742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.175084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 95.187130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 95.224129] Lustre: lustre-MDT0000: new disk, initializing [ 95.247167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.253893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 96.460872] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.528545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 101.559476] Lustre: 6462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.573164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.576380] Lustre: Skipped 1 previous similar message [ 101.612088] Lustre: lustre-MDT0001: new disk, initializing [ 101.633840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.641650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.646193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 102.870262] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.091863] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 108.228643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 108.315485] Lustre: lustre-OST0000: new disk, initializing [ 108.318199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 108.321667] Lustre: 8368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 108.339267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 110.087394] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.131084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 113.133927] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 113.156797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 115.296211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 115.337509] Lustre: lustre-OST0001: new disk, initializing [ 115.339063] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 115.341791] Lustre: 9426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 115.359244] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 117.222265] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.084123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.087867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.101200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 123.047343] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 127.765639] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 129.806930] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing check_logdir /tmp/testlogs/ [ 131.388807] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing yml_node [ 132.861699] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 133.684280] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 134.489602] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 135.025811] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 3 20:35:41 EDT 2026 [ 141.014862] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 141.512380] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.104833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.105265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.109205] Lustre: Skipped 3 previous similar messages [ 155.110929] Lustre: Skipped 2 previous similar messages [ 159.381842] Lustre: server umount lustre-MDT0000 complete [ 160.526549] LustreError: 9425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533367 with bad export cookie 15934477449772688068 [ 160.527762] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 160.530203] LustreError: 9425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 165.344769] LustreError: 6470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 165.344977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.345483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.345488] Lustre: Skipped 1 previous similar message [ 165.353945] LustreError: 6470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 165.359691] Lustre: Skipped 1 previous similar message [ 166.696914] Lustre: server umount lustre-MDT0001 complete [ 182.240137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 182.270497] Lustre: server umount lustre-OST0000 complete [ 189.649445] Lustre: server umount lustre-OST0001 complete [ 191.842121] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 194.782289] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 205.766193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.846866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 205.859353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 205.894115] Lustre: lustre-MDT0000: new disk, initializing [ 205.917308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.924988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.194771] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 211.155054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 211.193921] Lustre: 14935:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.288037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 212.467683] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 215.412223] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 216.633564] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 218.895338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.016554] Lustre: lustre-OST0000: new disk, initializing [ 219.019126] Lustre: Skipped 1 previous similar message [ 219.021185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.025211] Lustre: Skipped 2 previous similar messages [ 219.028797] Lustre: 16679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.066625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.068939] Lustre: Skipped 1 previous similar message [ 220.223539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.227256] Lustre: Skipped 1 previous similar message [ 220.230403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.281568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.063380] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 224.218786] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 225.746887] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 225.826207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 227.016451] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 227.082359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 230.369018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 230.371093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.374846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 230.376438] Lustre: Skipped 1 previous similar message [ 230.885158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 233.747212] Lustre: server umount lustre-OST0000 complete [ 236.001528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.002089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 236.005128] Lustre: Skipped 2 previous similar messages [ 236.006652] Lustre: Skipped 1 previous similar message [ 241.121653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 241.124251] Lustre: Skipped 2 previous similar messages [ 249.312086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 249.379599] Lustre: server umount lustre-MDT0000 complete [ 250.653280] LustreError: 15726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533457 with bad export cookie 15934477449772689587 [ 250.656653] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.656659] LustreError: 15726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 250.789048] Lustre: server umount lustre-MDT0001 complete [ 251.951936] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:37:38 (1780533458) [ 254.280628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 254.455279] LustreError: 18796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.481728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.812303] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 258.619844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.073350] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 261.891696] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 263.075265] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 265.158255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.033969] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 268.696095] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 289.766583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 289.766657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 289.773691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 289.776505] Lustre: Skipped 3 previous similar messages [ 293.257934] Lustre: server umount lustre-OST0000 complete [ 294.882679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 294.885151] LustreError: Skipped 1 previous similar message [ 300.503390] Lustre: server umount lustre-MDT0000 complete [ 301.747423] LustreError: 18777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533509 with bad export cookie 15934477449772690469 [ 301.751643] LustreError: 18777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 301.752191] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.885322] Lustre: server umount lustre-MDT0001 complete [ 305.455696] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 306.501969] Key type lgssc unregistered [ 306.633405] LNet: 21995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.636675] LNetError: 21995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.648284] LNet: Removed LNI 192.168.204.102@tcp [ 306.999107] Key type .llcrypt unregistered [ 307.000252] Key type ._llcrypt unregistered [ 315.430296] Key type ._llcrypt registered [ 315.432127] Key type .llcrypt registered [ 315.466695] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 315.888433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 315.900358] alg: No test for adler32 (adler32-zlib) [ 316.749975] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 316.834406] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 318.416139] Key type lgssc registered [ 318.752982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 321.290195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 321.300197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.498587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.808537] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 326.545697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 326.684037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 328.036476] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 329.756605] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 330.921273] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 333.144962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 333.271387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.109176] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 336.922559] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 348.649204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 351.713037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 351.716234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.721052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 353.763755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.763983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.764565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 357.525653] Lustre: server umount lustre-OST0000 complete [ 358.881102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.881833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.886487] Lustre: Skipped 1 previous similar message [ 358.890512] Lustre: Skipped 2 previous similar messages [ 364.000897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.003113] Lustre: Skipped 1 previous similar message [ 369.120964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 369.122688] Lustre: Skipped 1 previous similar message [ 373.216186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 373.282783] Lustre: server umount lustre-MDT0000 complete [ 374.241106] LustreError: 23368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.245864] LustreError: 23368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 374.482090] LustreError: 24982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533581 with bad export cookie 4411453913896797966 [ 374.485118] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.486192] LustreError: 24982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 374.598407] Lustre: server umount lustre-MDT0001 complete [ 377.774877] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 378.675313] Key type lgssc unregistered [ 378.805375] LNet: 27006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.808113] LNetError: 27006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.814269] LNet: Removed LNI 192.168.204.102@tcp [ 379.089235] Key type .llcrypt unregistered [ 379.090463] Key type ._llcrypt unregistered [ 387.123093] Key type ._llcrypt registered [ 387.124173] Key type .llcrypt registered [ 387.156608] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 387.520380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.532270] alg: No test for adler32 (adler32-zlib) [ 388.377658] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 388.454886] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 390.032074] Key type lgssc registered [ 390.356890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 392.650663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 392.656513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.801308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.108993] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 397.910861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.042353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.425240] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 401.217991] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 402.438268] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 406.934472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 407.072041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.033179] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 410.723280] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.367415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 423.392888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.395407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.399521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.418671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.420189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 424.424028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 424.425575] Lustre: Skipped 1 previous similar message [ 428.624299] Lustre: server umount lustre-OST0000 complete [ 434.144768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 434.147188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.152216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 439.777107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 439.778920] Lustre: Skipped 2 previous similar messages [ 444.384077] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.449360] Lustre: server umount lustre-MDT0000 complete [ 444.897032] LustreError: 28389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.902480] LustreError: 28389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 445.762187] LustreError: 28368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533653 with bad export cookie 2912261266927691488 [ 445.764787] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 445.767509] LustreError: 28368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 445.897204] Lustre: server umount lustre-MDT0001 complete [ 449.502430] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 450.509549] Key type lgssc unregistered [ 450.646401] LNet: 31931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.650041] LNetError: 31931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.659328] LNet: Removed LNI 192.168.204.102@tcp [ 450.968167] Key type .llcrypt unregistered [ 450.969666] Key type ._llcrypt unregistered [ 459.324996] Key type ._llcrypt registered [ 459.326372] Key type .llcrypt registered [ 459.364479] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 459.701095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 459.732085] alg: No test for adler32 (adler32-zlib) [ 460.593483] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 460.680487] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 462.264171] Key type lgssc registered [ 462.616575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 465.055455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 465.062138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.218599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.447355] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 469.996928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.110034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.256519] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 472.879076] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 473.913613] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 475.978776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.107271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.159685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 477.838058] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 479.516976] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.346483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.346913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.349152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.351791] Lustre: Skipped 1 previous similar message [ 489.108265] Lustre: server umount lustre-OST0000 complete [ 490.465497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 490.465515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.466102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.466106] Lustre: Skipped 1 previous similar message [ 490.475031] Lustre: Skipped 1 previous similar message [ 495.585042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.587584] Lustre: Skipped 2 previous similar messages [ 496.472120] Lustre: server umount lustre-MDT0000 complete [ 497.623534] LustreError: 33284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533704 with bad export cookie 15245967954755189115 [ 497.627540] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.627967] LustreError: 33284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 497.737936] Lustre: server umount lustre-MDT0001 complete [ 501.166793] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 502.180035] Key type lgssc unregistered [ 502.301357] LNet: 36483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.303901] LNetError: 36483:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.319316] LNet: Removed LNI 192.168.204.102@tcp [ 502.616822] Key type .llcrypt unregistered [ 502.618230] Key type ._llcrypt unregistered [ 510.625551] Key type ._llcrypt registered [ 510.626976] Key type .llcrypt registered [ 510.657597] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 510.993183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 511.028788] alg: No test for adler32 (adler32-zlib) [ 511.868559] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 511.937355] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 513.512097] Key type lgssc registered [ 513.840967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 516.194964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 516.200848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.352474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.552822] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 521.202043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.316277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 522.513704] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 524.139088] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 525.265951] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 527.422363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.532474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.553606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 529.313140] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 531.051695] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 537.058406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.060730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.065780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.067497] Lustre: Skipped 1 previous similar message [ 540.306484] Lustre: server umount lustre-OST0000 complete [ 553.010655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.011605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 553.016471] Lustre: Skipped 2 previous similar messages [ 553.019256] Lustre: Skipped 1 previous similar message [ 557.153404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.156525] Lustre: Skipped 2 previous similar messages [ 559.190411] Lustre: server umount lustre-MDT0000 complete [ 560.408133] LustreError: 37834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533767 with bad export cookie 8141260784225437873 [ 560.410274] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.412062] LustreError: 37834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 566.571566] Lustre: server umount lustre-MDT0001 complete [ 569.994973] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 570.990910] Key type lgssc unregistered [ 571.108380] LNet: 41205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.111617] LNetError: 41205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.120271] LNet: Removed LNI 192.168.204.102@tcp [ 571.397476] Key type .llcrypt unregistered [ 571.399255] Key type ._llcrypt unregistered [ 579.697986] Key type ._llcrypt registered [ 579.699510] Key type .llcrypt registered [ 579.733397] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 580.043743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 580.107176] alg: No test for adler32 (adler32-zlib) [ 580.955623] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 581.048025] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 582.632180] Key type lgssc registered [ 582.991703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.534283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 585.543857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.701655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 587.975168] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 590.707897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 590.823097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 592.018203] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 593.654187] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.771720] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 596.976904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.089114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.928273] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 600.493389] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 611.430086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 621.424127] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.2@tcp (stopping) [ 621.536765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.539308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.050173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.051300] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.053484] Lustre: Skipped 2 previous similar messages [ 622.057255] Lustre: Skipped 1 previous similar message [ 624.723707] Lustre: server umount lustre-MDT0000 complete [ 625.847198] LustreError: 42558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780533833 with bad export cookie 9933170988624449274 [ 625.851489] LustreError: 42558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 625.852309] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.535991] LustreError: 43291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.541170] LustreError: 43291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 626.541851] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.2@tcp (stopping) [ 626.545805] Lustre: Skipped 1 previous similar message [ 627.168760] LustreError: 42573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.168969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.176789] LustreError: 44728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.176983] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 630.184248] Lustre: Skipped 1 previous similar message [ 632.029622] Lustre: server umount lustre-MDT0001 complete [ 649.745455] Lustre: server umount lustre-OST0000 complete [ 654.579122] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 655.479115] Key type lgssc unregistered [ 655.591339] LNet: 46104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.593861] LNetError: 46104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.605290] LNet: Removed LNI 192.168.204.102@tcp [ 655.863287] Key type .llcrypt unregistered [ 655.864526] Key type ._llcrypt unregistered [ 665.026504] Key type ._llcrypt registered [ 665.027505] Key type .llcrypt registered [ 665.060902] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 665.338402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.346244] alg: No test for adler32 (adler32-zlib) [ 666.201437] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 666.280723] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 667.856117] Key type lgssc registered [ 668.186785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 670.582557] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 670.589349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 686.048343] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 691.168198] LustreError: 47454:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 717.792125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.492041] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 721.295918] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 887.019399] Lustre: server umount lustre-OST0000 complete [ 892.286921] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 893.264775] Key type lgssc unregistered [ 893.384367] LNet: 48997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.386970] LNetError: 48997:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.397273] LNet: Removed LNI 192.168.204.102@tcp [ 893.662499] Key type .llcrypt unregistered [ 893.663608] Key type ._llcrypt unregistered [ 901.840044] Key type ._llcrypt registered [ 901.841539] Key type .llcrypt registered [ 901.878506] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 902.153198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.167221] alg: No test for adler32 (adler32-zlib) [ 903.021016] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 903.100773] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 904.680100] Key type lgssc registered [ 905.034544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 907.419716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 907.424751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.570235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.773545] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 912.348920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 912.451045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.545356] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 915.077627] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 916.185423] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 918.169283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.273746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.333616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 919.918360] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 921.489122] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 923.233912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.234969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.238843] Lustre: Skipped 1 previous similar message [ 928.266954] Lustre: server umount lustre-OST0000 complete [ 932.833020] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.835467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.839243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.857090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.857431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 935.568314] Lustre: server umount lustre-MDT0000 complete [ 936.675104] LustreError: 50353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534143 with bad export cookie 2946617382480053022 [ 936.678026] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.679206] LustreError: 50353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 936.781264] Lustre: server umount lustre-MDT0001 complete [ 939.835153] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 940.733579] Key type lgssc unregistered [ 940.852353] LNet: 53549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.854834] LNetError: 53549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.862227] LNet: Removed LNI 192.168.204.102@tcp [ 941.125305] Key type .llcrypt unregistered [ 941.126265] Key type ._llcrypt unregistered [ 949.077099] Key type ._llcrypt registered [ 949.078177] Key type .llcrypt registered [ 949.114792] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 949.608776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 949.622676] alg: No test for adler32 (adler32-zlib) [ 950.475108] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 950.554942] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 952.136096] Key type lgssc registered [ 952.433206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 954.806844] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 954.813833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.272381] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 975.392198] LustreError: 54897:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 1002.016149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.876993] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1005.590675] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1007.721201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.881141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.900981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1009.094077] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1011.842266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.965822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1013.179808] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1015.049247] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1016.316966] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1017.316207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1017.318740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.322092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1017.324668] Lustre: Skipped 1 previous similar message [ 1019.092476] Lustre: server umount lustre-OST0000 complete [ 1033.180942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.182339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.187418] Lustre: Skipped 1 previous similar message [ 1033.189367] Lustre: Skipped 1 previous similar message [ 1037.281490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.283564] Lustre: Skipped 2 previous similar messages [ 1039.454206] Lustre: server umount lustre-MDT0000 complete [ 1040.811335] LustreError: 54905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534248 with bad export cookie 17120634061933938904 [ 1040.814353] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.816065] LustreError: 54905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1055.200169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1055.296474] Lustre: server umount lustre-MDT0001 complete [ 1059.048890] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1060.159949] Key type lgssc unregistered [ 1060.304341] LNet: 58286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.307045] LNetError: 58286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.315505] LNet: Removed LNI 192.168.204.102@tcp [ 1060.630187] Key type .llcrypt unregistered [ 1060.631242] Key type ._llcrypt unregistered [ 1069.823247] Key type ._llcrypt registered [ 1069.824435] Key type .llcrypt registered [ 1069.863666] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1070.289769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.328071] alg: No test for adler32 (adler32-zlib) [ 1071.211618] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1071.311099] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1072.896173] Key type lgssc registered [ 1073.309406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1076.015576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1076.022071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.202630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.587874] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1081.328921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.452521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1082.796364] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1084.717711] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1086.070387] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1088.533881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1088.646354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.565702] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1092.238857] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1103.015926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1111.520798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1111.523226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.527204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.573750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.573926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1113.574210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.580825] Lustre: Skipped 1 previous similar message [ 1115.027170] Lustre: server umount lustre-OST0000 complete [ 1117.153181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1117.155559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.159450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.456664] Lustre: server umount lustre-MDT0000 complete [ 1123.696492] LustreError: 59640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534330 with bad export cookie 9279793102639634760 [ 1123.701873] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.702274] LustreError: 59640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1123.820332] Lustre: server umount lustre-MDT0001 complete [ 1127.308256] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1128.432264] Key type lgssc unregistered [ 1128.559404] LNet: 62855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.561850] LNetError: 62855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.573289] LNet: Removed LNI 192.168.204.102@tcp [ 1128.879901] Key type .llcrypt unregistered [ 1128.881078] Key type ._llcrypt unregistered [ 1138.418673] Key type ._llcrypt registered [ 1138.419715] Key type .llcrypt registered [ 1138.458031] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1138.840431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1138.861319] alg: No test for adler32 (adler32-zlib) [ 1139.720732] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1139.805792] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1141.384142] Key type lgssc registered [ 1141.741880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.629601] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:52:31 (1780534351) [ 1146.780779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1146.786396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.937769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.333911] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1152.699277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.857478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.177137] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1155.903848] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1157.017861] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1159.323396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.509372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.391800] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1163.213076] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1172.841946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1176.011835] Lustre: Failing over lustre-MDT0000 [ 1176.179255] Lustre: server umount lustre-MDT0000 complete [ 1178.081980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.083821] LustreError: 64518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.085673] Lustre: Skipped 2 previous similar messages [ 1178.090506] LustreError: 64518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1178.989531] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.993875] LustreError: 64514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1179.070472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.111231] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.174778] Lustre: *** cfs_fail_loc=135, val=0*** [ 1179.218071] LustreError: 67286:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1179.220705] LustreError: 67286:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.102@tcp: cfg command failed: rc = -2 [ 1179.224464] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1179.224464] [ 1179.230199] LustreError: MGC192.168.204.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1179.239237] LustreError: 67274:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1179.244674] LustreError: 67274:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1179.249565] LustreError: 67274:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1179.253097] LustreError: 67274:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1179.264487] Lustre: server umount lustre-MDT0000 complete [ 1179.266198] LustreError: 67274:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1181.446439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.483950] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1181.573324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.597901] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1182.770113] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1184.104526] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1186.791609] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1186.801345] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1186.816132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1189.345279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1189.349530] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.356279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.907335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.907588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.909637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.879773] Lustre: server umount lustre-OST0000 complete [ 1197.025878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1197.026223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.033561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1197.035473] Lustre: Skipped 2 previous similar messages [ 1201.429430] Lustre: server umount lustre-MDT0000 complete [ 1202.145073] LustreError: 65891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.150034] LustreError: 65891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1202.609953] LustreError: 64498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534409 with bad export cookie 16765418118186496624 [ 1202.613383] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.613488] LustreError: 64498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1202.718356] Lustre: server umount lustre-MDT0001 complete [ 1206.084813] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1207.039732] Key type lgssc unregistered [ 1207.158335] LNet: 69028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.160804] LNetError: 69028:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.173250] LNet: Removed LNI 192.168.204.102@tcp [ 1207.440484] Key type .llcrypt unregistered [ 1207.441692] Key type ._llcrypt unregistered [ 1215.733501] Key type ._llcrypt registered [ 1215.734704] Key type .llcrypt registered [ 1215.772848] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1216.109115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1216.166222] alg: No test for adler32 (adler32-zlib) [ 1217.016188] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1217.104182] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1218.688112] Key type lgssc registered [ 1219.048770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.463864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.469388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.617665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.979464] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1226.827245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.950942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1228.210445] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1229.857725] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1230.997307] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1233.079713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1233.211447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.057090] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1236.771661] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1247.528910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1254.339652] Lustre: Failing over lustre-MDT0000 [ 1254.591049] Lustre: server umount lustre-MDT0000 complete [ 1257.326348] LustreError: 70398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.331072] LustreError: 70398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1257.840173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.881382] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.942346] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1257.954965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1257.956584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1257.959550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1257.961070] Lustre: Skipped 2 previous similar messages [ 1257.962789] Lustre: Skipped 1 previous similar message [ 1258.004249] LustreError: 73173:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1258.007372] LustreError: 73173:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.102@tcp: cfg command failed: rc = -9 [ 1258.010600] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1258.010600] [ 1258.014065] LustreError: MGC192.168.204.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1258.018085] LustreError: 73161:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1258.020386] LustreError: 73161:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1258.023304] LustreError: 73161:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1258.025162] LustreError: 73161:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1258.038946] Lustre: server umount lustre-MDT0000 complete [ 1258.040241] LustreError: 73161:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1260.514254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.557015] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.662927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1260.685894] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1262.004383] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1262.440352] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1266.150380] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1266.161955] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1266.180250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1268.193452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.195914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.200167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.201952] Lustre: Skipped 1 previous similar message [ 1271.267159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1271.270913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.271907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1271.277176] Lustre: Skipped 1 previous similar message [ 1273.300550] Lustre: server umount lustre-OST0000 complete [ 1276.387856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1276.388268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.388811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1276.395610] Lustre: Skipped 1 previous similar message [ 1281.505048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.507880] Lustre: Skipped 2 previous similar messages [ 1289.184108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1289.246146] Lustre: server umount lustre-MDT0000 complete [ 1290.503370] LustreError: 70383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534497 with bad export cookie 2049468544091688286 [ 1290.507442] LustreError: 70383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1290.507471] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.619705] Lustre: server umount lustre-MDT0001 complete [ 1294.329102] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1295.449679] Key type lgssc unregistered [ 1295.586387] LNet: 74922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.588986] LNetError: 74922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.600465] LNet: Removed LNI 192.168.204.102@tcp [ 1295.908474] Key type .llcrypt unregistered [ 1295.909676] Key type ._llcrypt unregistered [ 1300.169790] Key type ._llcrypt registered [ 1300.170878] Key type .llcrypt registered [ 1300.211504] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1300.315328] Key type .llcrypt unregistered [ 1300.316429] Key type ._llcrypt unregistered [ 1308.578815] Key type ._llcrypt registered [ 1308.579927] Key type .llcrypt registered [ 1308.624780] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1309.011830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1309.029460] alg: No test for adler32 (adler32-zlib) [ 1309.907776] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1310.000292] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1311.584192] Key type lgssc registered [ 1311.943657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1314.375190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1314.380328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.518641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.710930] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1319.371566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1319.504389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1320.749711] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1322.475805] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1323.741964] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1325.901857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1326.011519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.731801] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1329.466851] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1340.326797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 1349.600961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1349.604880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.610555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.625432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1350.625968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1350.626367] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1353.676550] Lustre: server umount lustre-OST0000 complete [ 1355.233032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1355.236304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1355.240444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.242491] Lustre: Skipped 1 previous similar message [ 1360.865136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1360.867830] Lustre: Skipped 2 previous similar messages [ 1361.059439] Lustre: server umount lustre-MDT0000 complete [ 1362.323216] LustreError: 78384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534569 with bad export cookie 16885375209447221490 [ 1362.327102] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.327457] LustreError: 78384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1362.461893] Lustre: server umount lustre-MDT0001 complete [ 1366.027430] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1367.115446] Key type lgssc unregistered [ 1367.252585] LNet: 79965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.254762] LNetError: 79965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.267450] LNet: Removed LNI 192.168.204.102@tcp [ 1367.577130] Key type .llcrypt unregistered [ 1367.578206] Key type ._llcrypt unregistered [ 1376.693351] Key type ._llcrypt registered [ 1376.694552] Key type .llcrypt registered [ 1376.739128] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1377.224589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1377.250544] alg: No test for adler32 (adler32-zlib) [ 1378.114691] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1378.206675] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1379.792166] Key type lgssc registered [ 1380.215911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1382.765928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1382.772138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.929299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.282703] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1388.223730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.353364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1389.731556] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1391.582542] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1392.835676] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.110174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.219850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.046086] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1398.974200] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1409.575673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 1418.720967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.724056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.728601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.746141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.746642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.746777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.747931] Lustre: Skipped 1 previous similar message [ 1422.796287] Lustre: server umount lustre-OST0000 complete [ 1424.353245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1424.358152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.366106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.985244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.987712] Lustre: Skipped 2 previous similar messages [ 1430.293405] Lustre: server umount lustre-MDT0000 complete [ 1431.486056] LustreError: 81326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534638 with bad export cookie 13599725045912109270 [ 1431.488958] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1431.489851] LustreError: 81326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1431.611091] Lustre: server umount lustre-MDT0001 complete [ 1435.002364] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1436.005921] Key type lgssc unregistered [ 1436.125348] LNet: 84545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.127663] LNetError: 84545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.136277] LNet: Removed LNI 192.168.204.102@tcp [ 1436.442542] Key type .llcrypt unregistered [ 1436.443741] Key type ._llcrypt unregistered [ 1445.115566] Key type ._llcrypt registered [ 1445.116603] Key type .llcrypt registered [ 1445.157830] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1445.546966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1445.570881] alg: No test for adler32 (adler32-zlib) [ 1446.430707] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1446.525713] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1448.112161] Key type lgssc registered [ 1448.522646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1451.329875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1451.336973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.497084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.828468] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1456.584149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.704258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1458.012854] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1459.771654] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1461.025636] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1463.272116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1463.412581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.225362] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1466.949145] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1477.733782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 1487.330351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1487.333757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.339193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.354455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1488.356877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.357297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1488.363265] Lustre: Skipped 1 previous similar message [ 1491.858176] Lustre: server umount lustre-OST0000 complete [ 1493.473767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.474951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.480972] Lustre: Skipped 1 previous similar message [ 1498.593776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.597252] Lustre: Skipped 2 previous similar messages [ 1499.293838] Lustre: server umount lustre-MDT0000 complete [ 1500.532552] LustreError: 85900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534707 with bad export cookie 5184549345917856204 [ 1500.536859] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.539058] LustreError: 85900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1500.659971] Lustre: server umount lustre-MDT0001 complete [ 1504.109642] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1505.191184] Key type lgssc unregistered [ 1505.323372] LNet: 89115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.325564] LNetError: 89115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.336304] LNet: Removed LNI 192.168.204.102@tcp [ 1505.626766] Key type .llcrypt unregistered [ 1505.627973] Key type ._llcrypt unregistered [ 1514.076137] Key type ._llcrypt registered [ 1514.077288] Key type .llcrypt registered [ 1514.113766] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1514.407391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1514.443054] alg: No test for adler32 (adler32-zlib) [ 1515.302166] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1515.390053] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1516.977770] Key type lgssc registered [ 1517.363123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1519.847875] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1519.855091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.264328] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1540.385588] LustreError: 90462:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 1567.008141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.811181] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1570.541878] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1572.396944] Lustre: server umount lustre-OST0000 complete [ 1575.067330] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:59:41 (1780534781) [ 1577.226865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.383263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1578.557785] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1581.164128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.278394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.444705] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1584.033803] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1585.085553] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1587.170857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.283039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.327215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 1588.894677] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1590.472938] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1596.901045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1596.905046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1596.905262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1596.909511] Lustre: Skipped 1 previous similar message [ 1600.716288] Lustre: server umount lustre-OST0000 complete [ 1602.017949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1602.018148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1602.018846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1602.030017] Lustre: Skipped 2 previous similar messages [ 1607.136991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1607.139218] Lustre: Skipped 2 previous similar messages [ 1612.257765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1612.259717] Lustre: Skipped 1 previous similar message [ 1616.352163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1616.416339] Lustre: server umount lustre-MDT0000 complete [ 1617.378350] LustreError: 91986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.387725] LustreError: 91986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1617.633412] LustreError: 91966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534824 with bad export cookie 18096220108404908940 [ 1617.636356] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1617.636498] LustreError: 91966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1617.767639] Lustre: server umount lustre-MDT0001 complete [ 1621.168790] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1622.024853] Key type lgssc unregistered [ 1622.142336] LNet: 95323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.144714] LNetError: 95323:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.154274] LNet: Removed LNI 192.168.204.102@tcp [ 1622.405394] Key type .llcrypt unregistered [ 1622.406464] Key type ._llcrypt unregistered [ 1636.591871] Key type ._llcrypt registered [ 1636.592926] Key type .llcrypt registered [ 1636.629420] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1637.092259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1637.115209] alg: No test for adler32 (adler32-zlib) [ 1637.976046] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1638.059060] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1639.640101] Key type lgssc registered [ 1639.957093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1642.257598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1642.263213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.413018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.499916] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1647.021940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.146655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1648.395600] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1650.094319] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1651.142616] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1653.140065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1653.217906] LustreError: 99059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1653.222321] LustreError: 99059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1653.249142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.882653] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1656.444527] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1657.383763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 1667.556277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.556813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1667.560259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1667.561566] LustreError: Skipped 1 previous similar message [ 1667.563956] Lustre: Skipped 1 previous similar message [ 1671.241310] Lustre: server umount lustre-OST0000 complete [ 1672.673475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1672.673512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.678864] Lustre: Skipped 1 previous similar message [ 1672.680625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.682294] Lustre: Skipped 1 previous similar message [ 1677.793467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1677.795383] Lustre: Skipped 1 previous similar message [ 1678.485589] Lustre: server umount lustre-MDT0000 complete [ 1679.634249] LustreError: 97448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780534886 with bad export cookie 10355975213247601324 [ 1679.637807] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.640415] LustreError: 97448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1679.746564] Lustre: server umount lustre-MDT0001 complete [ 1682.892845] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1683.829227] Key type lgssc unregistered [ 1683.943338] LNet: 100659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.945619] LNetError: 100659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.955298] LNet: Removed LNI 192.168.204.102@tcp [ 1684.230537] Key type .llcrypt unregistered [ 1684.231494] Key type ._llcrypt unregistered [ 1690.954277] Key type ._llcrypt registered [ 1690.955248] Key type .llcrypt registered [ 1690.994793] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1691.340945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1691.359909] alg: No test for adler32 (adler32-zlib) [ 1692.204496] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1692.286413] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1693.864164] Key type lgssc registered [ 1694.196680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1697.478878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1697.490536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.992495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1718.112195] LustreError: 101862:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 1744.736155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.469805] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1748.493528] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1750.756469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.840772] LustreError: MGC192.168.204.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1750.844568] LustreError: 102772:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1750.846792] LustreError: 102772:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1750.849161] LustreError: 102772:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1750.874694] Lustre: server umount lustre-MDT0000 complete [ 1750.875961] LustreError: 102772:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1763.334670] Lustre: server umount lustre-OST0000 complete [ 1766.114137] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1769.059343] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1779.897315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.981833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1779.991921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1780.027153] Lustre: lustre-MDT0000: new disk, initializing [ 1780.048218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.053599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1781.230605] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1785.521081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.564331] Lustre: 105525:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1785.588410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1785.590575] Lustre: Skipped 1 previous similar message [ 1785.628408] Lustre: lustre-MDT0001: new disk, initializing [ 1785.663698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.676069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1785.679516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1787.076934] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1790.275169] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1791.598243] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1793.965411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.078655] Lustre: lustre-OST0000: new disk, initializing [ 1794.080316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1794.082729] Lustre: 107272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1794.119938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.612334] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1798.648321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1798.651693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1798.682939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1799.948389] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1801.277125] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1801.346723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1802.738778] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1802.816654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1803.746477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.747384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.749912] LustreError: Skipped 1 previous similar message [ 1803.756935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1803.760684] Lustre: Skipped 1 previous similar message [ 1805.728680] Lustre: server umount lustre-OST0000 complete [ 1808.865841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.866394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.869641] Lustre: Skipped 2 previous similar messages [ 1813.218546] Lustre: server umount lustre-MDT0000 complete [ 1814.603821] LustreError: 105519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535021 with bad export cookie 3096728599752534446 [ 1814.607655] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.608892] LustreError: 105519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1814.759125] Lustre: server umount lustre-MDT0001 complete [ 1817.760225] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:03:44 (1780535024) [ 1819.256382] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 1826.501310] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1830.498277] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1844.455039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.549456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.559906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1844.596932] Lustre: lustre-MDT0000: new disk, initializing [ 1844.618626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1844.624226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1846.056184] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1851.505927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1851.553540] Lustre: 111984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1851.584394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1851.588313] Lustre: Skipped 1 previous similar message [ 1851.639192] Lustre: lustre-MDT0001: new disk, initializing [ 1851.703461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1851.710973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1853.391552] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1857.440545] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.107647] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1861.644597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1861.749345] Lustre: lustre-OST0000: new disk, initializing [ 1861.752101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1861.754582] Lustre: 113726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1861.785670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1861.788397] Lustre: Skipped 1 previous similar message [ 1863.614280] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1867.247628] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1868.781954] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1870.353610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1870.361651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1870.410306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1870.954239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1872.699128] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1872.809456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1875.431359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.433240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.436539] LustreError: Skipped 1 previous similar message [ 1875.446202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.453655] Lustre: Skipped 2 previous similar messages [ 1879.784159] Lustre: server umount lustre-OST0000 complete [ 1882.593172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.596565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.601929] Lustre: Skipped 1 previous similar message [ 1882.604215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.690253] Lustre: server umount lustre-MDT0000 complete [ 1889.501376] LustreError: 111977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535096 with bad export cookie 3096728599752535321 [ 1889.508267] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1889.508492] LustreError: 111977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1889.665714] Lustre: server umount lustre-MDT0001 complete [ 1893.411614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.612915] LustreError: 115815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1893.622582] LustreError: 115815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1893.648851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.262384] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1898.977772] LustreError: 115816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.320748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.144104] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1903.495051] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1905.067757] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1907.916716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1910.229294] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1912.334143] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1926.031505] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 1928.675726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1928.677215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1928.680712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.687434] Lustre: Skipped 2 previous similar messages [ 1928.691554] Lustre: Skipped 2 previous similar messages [ 1933.302400] Lustre: server umount lustre-OST0000 complete [ 1935.329436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1938.914149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.915128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.918229] Lustre: Skipped 2 previous similar messages [ 1938.929661] Lustre: Skipped 1 previous similar message [ 1941.118104] Lustre: server umount lustre-MDT0000 complete [ 1942.924781] LustreError: 115795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535150 with bad export cookie 3096728599752536189 [ 1942.931288] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.933363] LustreError: 115795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1943.067981] Lustre: server umount lustre-MDT0001 complete [ 1947.606176] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 1948.888452] Key type lgssc unregistered [ 1949.056746] LNet: 119110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.062112] LNetError: 119110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.070724] LNet: Removed LNI 192.168.204.102@tcp [ 1949.545453] Key type .llcrypt unregistered [ 1949.548459] Key type ._llcrypt unregistered [ 1957.302859] Key type ._llcrypt registered [ 1957.304210] Key type .llcrypt registered [ 1957.345265] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 1964.015086] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 1964.379468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.386117] alg: No test for adler32 (adler32-zlib) [ 1965.241557] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 1965.335514] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 1966.936189] Key type lgssc registered [ 1967.474286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.888157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.897398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.002491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1983.019588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1983.060796] Lustre: lustre-MDT0000: new disk, initializing [ 1983.087652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.097099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1984.786362] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1990.018888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.069669] Lustre: 122203:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1990.106705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1990.109771] Lustre: Skipped 1 previous similar message [ 1990.155334] Lustre: lustre-MDT0001: new disk, initializing [ 1990.188360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.199231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1990.202350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1991.812710] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 1995.336333] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1997.035652] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2000.230267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.392405] Lustre: lustre-OST0000: new disk, initializing [ 2000.395209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2000.398673] Lustre: 123950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2000.435479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.425261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2001.430757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2001.476287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2002.842663] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2006.850464] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2008.633886] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2008.756645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2010.130301] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2010.201250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2011.622806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.623085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.628358] LustreError: Skipped 1 previous similar message [ 2011.638316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.643751] Lustre: Skipped 1 previous similar message [ 2016.741527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2016.745420] Lustre: Skipped 1 previous similar message [ 2017.006830] Lustre: server umount lustre-OST0000 complete [ 2020.833874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2020.840401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.849499] Lustre: Skipped 1 previous similar message [ 2020.855162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.859920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.626741] Lustre: server umount lustre-MDT0000 complete [ 2026.209585] LustreError: 123949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535233 with bad export cookie 8864646077939228657 [ 2026.214245] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2026.215385] LustreError: 123949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2026.345301] Lustre: server umount lustre-MDT0001 complete [ 2029.731051] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:07:16 (1780535236) [ 2032.106390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.288168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.873761] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2037.197900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.347920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2038.830057] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2041.302462] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2042.750764] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2047.457037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2047.460118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.465544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.468837] Lustre: Skipped 2 previous similar messages [ 2049.579634] Lustre: server umount lustre-MDT0000 complete [ 2051.221292] LustreError: 126196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535258 with bad export cookie 8864646077939229525 [ 2051.224788] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2051.227218] LustreError: 126196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2051.336750] Lustre: server umount lustre-MDT0001 complete [ 2054.491387] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:07:41 (1780535261) [ 2057.063592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2072.544665] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2077.664682] LustreError: 128404:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 2103.264561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2105.805546] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2107.954718] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2108.797839] Lustre: server umount lustre-OST0000 complete [ 2111.923736] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:08:38 (1780535318) [ 2114.534418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2114.739888] LustreError: 129734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.751713] LustreError: 129734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2116.141104] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2119.812015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2119.943655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.946688] Lustre: Skipped 1 previous similar message [ 2121.460437] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2123.376620] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.577636] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2126.792249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.763460] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2130.525130] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2145.764397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.767297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.770746] Lustre: Skipped 3 previous similar messages [ 2145.774399] Lustre: Skipped 4 previous similar messages [ 2145.778175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.506421] Lustre: server umount lustre-MDT0000 complete [ 2149.733332] LustreError: 129715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535357 with bad export cookie 8864646077939230176 [ 2149.736306] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.737675] LustreError: 129715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2149.850439] Lustre: server umount lustre-MDT0001 complete [ 2161.441650] Lustre: server umount lustre-OST0000 complete [ 2164.827279] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:09:31 (1780535371) [ 2167.735891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.897653] LustreError: 133069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.918895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.920903] Lustre: Skipped 1 previous similar message [ 2169.238250] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2172.456422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.138561] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2176.083674] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.542901] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2180.496100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.599203] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2184.304136] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2185.633649] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2193.961581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2195.002967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2196.698435] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2196.831963] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2197.985392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2197.990288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.998664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2198.003162] Lustre: Skipped 1 previous similar message [ 2203.797261] Lustre: server umount lustre-OST0000 complete [ 2208.737386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.741028] LustreError: Skipped 1 previous similar message [ 2211.352560] Lustre: server umount lustre-MDT0000 complete [ 2212.824381] LustreError: 133050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535420 with bad export cookie 8864646077939231233 [ 2212.828569] LustreError: 133050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2212.828990] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.970773] Lustre: server umount lustre-MDT0001 complete [ 2216.506296] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:10:23 (1780535423) [ 2219.315171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2234.720968] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2239.840725] LustreError: 136923:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 2265.440613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.445485] Lustre: Skipped 2 previous similar messages [ 2267.720525] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2269.667321] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2272.869394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.121767] LustreError: 136948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.195481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2274.932006] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2278.371732] LustreError: 136949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.139610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.112149] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2283.165554] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2284.812194] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2286.515571] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2286.596104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2288.271388] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2288.353073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2289.634434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.639381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.646841] Lustre: Skipped 3 previous similar messages [ 2289.650618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.654783] Lustre: Skipped 5 previous similar messages [ 2295.401755] Lustre: server umount lustre-MDT0000 complete [ 2295.780085] LustreError: 137848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.786814] LustreError: 137848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2296.952901] LustreError: 136929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535504 with bad export cookie 8864646077939232241 [ 2296.958807] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2296.961391] LustreError: 136929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2297.107313] Lustre: server umount lustre-MDT0001 complete [ 2307.831302] Lustre: server umount lustre-OST0000 complete [ 2311.239409] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:11:57 (1780535517) [ 2314.029461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.504786] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2334.624572] LustreError: 140746:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 2362.516166] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2364.555756] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2367.538555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.712762] LustreError: 140771:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2367.759092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 2368.920556] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2372.872857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.516579] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2376.988087] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2378.731829] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2381.842164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2381.947124] Lustre: lustre-OST0001: new disk, initializing [ 2381.949872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2381.953588] Lustre: 143280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2384.681924] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2388.702719] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2390.393460] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2392.078298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2392.085528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2392.144860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2392.596993] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2394.456447] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2394.589441] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2397.163831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.168134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2397.173377] Lustre: Skipped 3 previous similar messages [ 2397.177136] Lustre: Skipped 3 previous similar messages [ 2401.478277] Lustre: server umount lustre-OST0000 complete [ 2402.279655] LustreError: 142387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.286087] LustreError: 142387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2407.395106] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2407.399506] LustreError: Skipped 1 previous similar message [ 2409.075725] Lustre: server umount lustre-OST0001 complete [ 2416.890858] Lustre: server umount lustre-MDT0000 complete [ 2418.523766] LustreError: 140753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535625 with bad export cookie 8864646077939233193 [ 2418.529512] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.529731] LustreError: 140753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2418.661589] Lustre: server umount lustre-MDT0001 complete [ 2427.256972] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:13:53 (1780535633) [ 2428.029986] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 2428.694211] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 2429.253807] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:13:56 (1780535636) [ 2432.125449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.251199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2432.264454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2432.372634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.374714] Lustre: Skipped 6 previous similar messages [ 2433.938971] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2437.219067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.261658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2437.271667] Lustre: 146704:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2438.895598] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2441.162163] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2442.400422] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2444.779366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2444.865363] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2447.222737] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2448.627430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 2449.473551] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2451.286519] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2451.408906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2452.910527] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2453.037464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2464.226247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.227980] Lustre: Skipped 10 previous similar messages [ 2468.320121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2468.374759] Lustre: server umount lustre-OST0000 complete [ 2482.120541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2484.484187] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2487.111718] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2489.190718] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2494.567674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.573148] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.573310] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2494.581842] Lustre: Skipped 1 previous similar message [ 2495.444217] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 2496.668539] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2496.745501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2497.905688] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9af7c9a0f000.ost_server_uuid 50 [ 2498.585041] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9af7c9a0f000.ost_server_uuid in FULL state after 0 sec [ 2498.711124] LustreError: 146711:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.2@tcp) returned error from blocking AST (req@00000000662ffa16 x1867026448710656 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000011f10abd/0x7b058ca41720baf5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.2@tcp remote: 0x11ec32a05975593b expref: 5 pid: 146711 timeout: 2598 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2498.731423] LustreError: lustre-MDT0000: A client on nid 192.168.204.2@tcp was evicted due to a lock blocking callback time out: rc -107 [ 2498.737607] LustreError: 146699:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.2@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000011f10abd/0x7b058ca41720baf5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.2@tcp remote: 0x11ec32a05975593b expref: 6 pid: 146711 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 2500.067644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.067810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.069926] LustreError: Skipped 3 previous similar messages [ 2500.073366] Lustre: Skipped 6 previous similar messages [ 2513.888153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2513.971597] Lustre: server umount lustre-OST0000 complete [ 2530.272090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2530.340104] Lustre: server umount lustre-MDT0000 complete [ 2530.786857] LustreError: 146715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.797320] LustreError: 146715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2532.129685] LustreError: 147428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535739 with bad export cookie 8864646077939234376 [ 2532.137523] LustreError: 147428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2532.138623] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2538.360495] Lustre: server umount lustre-MDT0001 complete [ 2543.473926] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2544.728872] Key type lgssc unregistered [ 2544.843342] LNet: 152228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.845598] LNetError: 152228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2544.854293] LNet: Removed LNI 192.168.204.102@tcp [ 2545.139616] Key type .llcrypt unregistered [ 2545.141576] Key type ._llcrypt unregistered [ 2556.077959] Key type ._llcrypt registered [ 2556.079295] Key type .llcrypt registered [ 2556.114969] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2556.496236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.506497] alg: No test for adler32 (adler32-zlib) [ 2557.378722] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2557.466182] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2559.056199] Key type lgssc registered [ 2559.682809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2563.018324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2563.030080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.197191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.844064] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2569.810853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.955588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.447369] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2573.986596] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2575.601319] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2578.673959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.843333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.209623] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2583.432070] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2589.106767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2596.759578] Lustre: Failing over lustre-MDT0000 [ 2596.981232] Lustre: server umount lustre-MDT0000 complete [ 2599.397667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2599.400838] LustreError: 153596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.405577] Lustre: Skipped 2 previous similar messages [ 2599.414376] LustreError: 153596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2604.513809] LustreError: 153602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.525274] LustreError: 153602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2608.175500] LustreError: 154322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.2@tcp arrived at 1780535815 with bad export cookie 14418133536255896707 [ 2608.188422] LustreError: 154322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2609.636839] LustreError: 153601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.646599] LustreError: 153601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2611.354574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.408779] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.540203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.579705] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2613.021062] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2613.732316] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2613.737990] Lustre: lustre-MDT0000: Denying connection for new client 3c68538d-b76e-4ef6-bdf2-10abb5a44caa (at 192.168.204.2@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2616.806893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2620.897450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2620.902756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2620.911743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.922297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2621.927327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.725488] Lustre: server umount lustre-OST0000 complete [ 2628.481359] LustreError: 157000:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2628.486182] Lustre: 156365:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2628.491774] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2628.502162] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2628.509767] LustreError: 156365:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000475d90a9 x1867027069309824/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2628.520918] LustreError: 156365:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 2628.527166] LustreError: 156365:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 2628.533252] Lustre: lustre-MDT0000: Recovery over after 0:15, of 2 clients 0 recovered and 2 were evicted. [ 2628.666692] Lustre: server umount lustre-MDT0000 complete [ 2630.382788] LustreError: 153583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535837 with bad export cookie 14418133536255897064 [ 2630.388944] LustreError: 152912:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000009b7d8e4 x1867027069310720/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2630.389425] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.390066] LustreError: 153583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2636.626197] Lustre: server umount lustre-MDT0001 complete [ 2640.969238] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2642.197565] Key type lgssc unregistered [ 2642.340658] LNet: 157735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.343045] LNetError: 157735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.349714] LNet: Removed LNI 192.168.204.102@tcp [ 2642.710059] Key type .llcrypt unregistered [ 2642.711114] Key type ._llcrypt unregistered [ 2654.504828] Key type ._llcrypt registered [ 2654.505918] Key type .llcrypt registered [ 2654.559348] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2654.955214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2654.962160] alg: No test for adler32 (adler32-zlib) [ 2655.822799] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2655.936335] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2657.536257] Key type lgssc registered [ 2658.196993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2661.658735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2661.669402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.945451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.778866] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2668.943089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.127925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2671.074204] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2673.635332] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2675.291844] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2678.344863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.490319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.107530] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2683.508316] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2688.756247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2694.120482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.124073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.127048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.134566] Lustre: Skipped 1 previous similar message [ 2696.239476] Lustre: server umount lustre-OST0000 complete [ 2699.236353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.237639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.240860] Lustre: Skipped 1 previous similar message [ 2699.244460] Lustre: Skipped 1 previous similar message [ 2704.125752] Lustre: server umount lustre-MDT0000 complete [ 2704.355205] LustreError: 159109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.365668] LustreError: 159109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2706.034258] LustreError: 159089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535913 with bad export cookie 11699940093726227976 [ 2706.041760] LustreError: 159089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2706.042824] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.211872] Lustre: server umount lustre-MDT0001 complete [ 2711.369202] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2713.008605] Key type lgssc unregistered [ 2713.190661] LNet: 162285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2713.195918] LNetError: 162285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2713.207691] LNet: Removed LNI 192.168.204.102@tcp [ 2713.708209] Key type .llcrypt unregistered [ 2713.710586] Key type ._llcrypt unregistered [ 2729.174860] Key type ._llcrypt registered [ 2729.175884] Key type .llcrypt registered [ 2729.213714] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2729.582061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.637897] alg: No test for adler32 (adler32-zlib) [ 2730.536221] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2730.645245] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2732.232194] Key type lgssc registered [ 2732.858604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.076634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.090512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.300399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.977911] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2742.876221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.064833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.919392] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2747.452760] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2749.309495] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.633996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.788182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.238570] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2757.675565] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2762.032195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2767.336843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.337642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2767.339251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2767.344778] Lustre: Skipped 1 previous similar message [ 2770.606828] Lustre: server umount lustre-OST0000 complete [ 2772.453964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.463580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.468095] Lustre: Skipped 2 previous similar messages [ 2777.570144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2777.574104] Lustre: Skipped 1 previous similar message [ 2782.691362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2782.695231] Lustre: Skipped 1 previous similar message [ 2786.272165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2786.372867] Lustre: server umount lustre-MDT0000 complete [ 2786.786651] LustreError: 163946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2786.796169] LustreError: 163946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2788.146160] LustreError: 163926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780535995 with bad export cookie 2283065429770835377 [ 2788.152131] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2788.153462] LustreError: 163926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2788.319985] Lustre: server umount lustre-MDT0001 complete [ 2793.176252] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2794.401394] Key type lgssc unregistered [ 2794.556781] LNet: 167128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.562802] LNetError: 167128:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2794.573601] LNet: Removed LNI 192.168.204.102@tcp [ 2795.048299] Key type .llcrypt unregistered [ 2795.051119] Key type ._llcrypt unregistered [ 2804.896933] Key type ._llcrypt registered [ 2804.897988] Key type .llcrypt registered [ 2804.931093] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2805.246656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2805.258540] alg: No test for adler32 (adler32-zlib) [ 2806.171432] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2806.282411] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2807.888157] Key type lgssc registered [ 2808.493571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2812.977059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2812.987861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.164263] Lustre: *** cfs_fail_loc=135, val=0*** [ 2814.205901] LustreError: 168614:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2814.207914] LustreError: 168614:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.102@tcp: cfg command failed: rc = -2 [ 2814.210181] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2814.210181] [ 2814.212564] LustreError: MGC192.168.204.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2814.216422] LustreError: 168575:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2814.218786] LustreError: 168575:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2814.220944] LustreError: 168575:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2814.222871] LustreError: 168575:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2814.256115] Lustre: server umount lustre-MDT0000 complete [ 2814.257520] LustreError: 168575:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2817.105036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.321887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.111598] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2822.527113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2822.645477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2823.868705] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2826.237568] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2827.997121] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2831.842522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.843714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.848695] Lustre: Skipped 1 previous similar message [ 2832.866351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2832.870979] Lustre: Skipped 2 previous similar messages [ 2835.115095] Lustre: server umount lustre-MDT0000 complete [ 2836.874882] LustreError: 168966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536044 with bad export cookie 11903542353957822524 [ 2836.880774] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.882098] LustreError: 168966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2837.035186] Lustre: server umount lustre-MDT0001 complete [ 2841.258116] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2842.612384] Key type lgssc unregistered [ 2842.749484] LNet: 171065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2842.753380] LNetError: 171065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2842.770507] LNet: Removed LNI 192.168.204.102@tcp [ 2843.170803] Key type .llcrypt unregistered [ 2843.171808] Key type ._llcrypt unregistered [ 2852.069606] Key type ._llcrypt registered [ 2852.070888] Key type .llcrypt registered [ 2852.127155] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2852.283195] Key type .llcrypt unregistered [ 2852.284206] Key type ._llcrypt unregistered [ 2860.808773] Key type ._llcrypt registered [ 2860.810491] Key type .llcrypt registered [ 2860.867563] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2861.346844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2861.352938] alg: No test for adler32 (adler32-zlib) [ 2862.239363] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2862.346259] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2863.944229] Key type lgssc registered [ 2864.595941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2868.510499] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2868.522158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2884.000319] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2889.120386] LustreError: 173142:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.102@tcp: failed processing log, type 4: rc = -110 [ 2915.744199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.955672] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2920.409952] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2923.456046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.707543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.733445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2925.313347] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2929.121617] LustreError: 173168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2929.131789] LustreError: 173168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2929.468285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.644378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2931.237645] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 2933.731349] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2935.453233] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2936.795136] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 2949.601537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.608568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.620550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.114281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.119058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.127457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.140548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.146274] Lustre: Skipped 1 previous similar message [ 2956.260778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2956.263715] Lustre: Skipped 1 previous similar message [ 2961.378761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.383215] Lustre: Skipped 1 previous similar message [ 2962.400177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.481469] Lustre: server umount lustre-OST0000 complete [ 2965.473534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2965.478527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.354289] Lustre: server umount lustre-MDT0000 complete [ 2971.617933] LustreError: 174070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.628456] LustreError: 174070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2972.108327] LustreError: 174794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536179 with bad export cookie 12123754533153499013 [ 2972.113595] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.116867] LustreError: 174794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2972.271135] Lustre: server umount lustre-MDT0001 complete [ 2977.034565] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 2978.257992] Key type lgssc unregistered [ 2978.380462] LNet: 176899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2978.383336] LNetError: 176899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2978.392327] LNet: Removed LNI 192.168.204.102@tcp [ 2978.704493] Key type .llcrypt unregistered [ 2978.705626] Key type ._llcrypt unregistered [ 2990.014137] Key type ._llcrypt registered [ 2990.015939] Key type .llcrypt registered [ 2990.078006] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 2990.467680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2990.484602] alg: No test for adler32 (adler32-zlib) [ 2991.400651] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 2991.505897] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 2993.120248] Key type lgssc registered [ 2993.815300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.617766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.631103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.875197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.638308] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3004.033437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.204623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3005.886982] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3008.496090] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.166772] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.416858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3013.627086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.315753] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3019.024809] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.851220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3026.250535] Lustre: Failing over lustre-MDT0000 [ 3026.384524] Lustre: server umount lustre-MDT0000 complete [ 3028.966430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.969259] LustreError: 179481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.975544] Lustre: Skipped 2 previous similar messages [ 3028.985884] LustreError: 179481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3029.880103] LustreError: 179481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.891944] LustreError: 179481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3034.082356] LustreError: 179480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3034.092924] LustreError: 179480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3039.204373] LustreError: 178272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.214317] LustreError: 178272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3040.553695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.623841] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.759440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.792242] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3042.518441] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3044.812757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3045.226705] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3045.876514] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3045.888601] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3045.918734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3047.230144] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3053.027922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3053.032773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.041080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.585587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.590486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.599727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3058.925157] Lustre: server umount lustre-OST0000 complete [ 3061.221379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3061.222253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.223222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3061.223233] Lustre: Skipped 2 previous similar messages [ 3061.240214] Lustre: Skipped 1 previous similar message [ 3066.338488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3066.341710] Lustre: Skipped 2 previous similar messages [ 3071.458372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3071.462266] Lustre: Skipped 1 previous similar message [ 3075.040154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3075.131050] Lustre: server umount lustre-MDT0000 complete [ 3076.578325] LustreError: 178272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.588265] LustreError: 178272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3076.993902] LustreError: 178254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536284 with bad export cookie 523289392391389501 [ 3077.003438] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3077.004218] LustreError: 178254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3077.170800] Lustre: server umount lustre-MDT0001 complete [ 3082.315128] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3083.775464] Key type lgssc unregistered [ 3083.921615] LNet: 182686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.927157] LNetError: 182686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.935564] LNet: Removed LNI 192.168.204.102@tcp [ 3084.412865] Key type .llcrypt unregistered [ 3084.415629] Key type ._llcrypt unregistered [ 3095.602834] Key type ._llcrypt registered [ 3095.604050] Key type .llcrypt registered [ 3095.663530] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3096.237843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3096.247543] alg: No test for adler32 (adler32-zlib) [ 3097.135611] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3097.255215] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3098.872126] Key type lgssc registered [ 3099.222963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3102.507463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3102.519853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.726870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.484427] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3109.620261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.809937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3111.519993] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3114.165931] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3116.036731] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3119.316679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.517613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3121.721700] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3123.463046] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3128.748950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 3134.628712] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3143.527551] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 3155.351760] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3164.645173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3164.645441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.660737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3164.664636] Lustre: Skipped 1 previous similar message [ 3169.763309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3169.767040] Lustre: Skipped 1 previous similar message [ 3170.083802] Lustre: server umount lustre-OST0000 complete [ 3174.882951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.884453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.897305] Lustre: Skipped 2 previous similar messages [ 3180.001271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.003928] Lustre: Skipped 2 previous similar messages [ 3185.121843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3185.127568] Lustre: Skipped 1 previous similar message [ 3186.144170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3186.228321] Lustre: server umount lustre-MDT0000 complete [ 3188.027359] LustreError: 184039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536395 with bad export cookie 11844266379469104312 [ 3188.032633] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3188.037942] LustreError: 184039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3188.196389] Lustre: server umount lustre-MDT0001 complete [ 3193.105689] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3194.496079] Key type lgssc unregistered [ 3194.642559] LNet: 187578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3194.648198] LNetError: 187578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3194.657534] LNet: Removed LNI 192.168.204.102@tcp [ 3195.103563] Key type .llcrypt unregistered [ 3195.105795] Key type ._llcrypt unregistered [ 3204.217532] Key type ._llcrypt registered [ 3204.218665] Key type .llcrypt registered [ 3204.262546] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3204.560583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3204.567085] alg: No test for adler32 (adler32-zlib) [ 3205.433196] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3205.515148] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3207.096157] Key type lgssc registered [ 3207.694750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3211.433175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3211.438382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.651975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.349118] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3218.420923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3218.603844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3220.027749] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3222.373954] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.098934] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.249165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.409956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.774954] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3232.437290] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3237.675902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 3239.541436] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 3250.482831] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3256.754604] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 3265.634091] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 3273.703135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.703763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.712687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.721700] Lustre: Skipped 1 previous similar message [ 3277.730781] Lustre: server umount lustre-OST0000 complete [ 3280.353115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.355351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.358705] Lustre: Skipped 1 previous similar message [ 3280.360498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.938409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.939312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3285.493676] Lustre: server umount lustre-MDT0000 complete [ 3287.265167] LustreError: 188786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536494 with bad export cookie 1756054903133052901 [ 3287.272355] LustreError: 188786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3287.272597] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3287.414118] Lustre: server umount lustre-MDT0001 complete [ 3292.282594] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3293.773270] Key type lgssc unregistered [ 3293.934626] LNet: 193761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3293.939885] LNetError: 193761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3293.949556] LNet: Removed LNI 192.168.204.102@tcp [ 3294.399798] Key type .llcrypt unregistered [ 3294.401975] Key type ._llcrypt unregistered [ 3304.868870] Key type ._llcrypt registered [ 3304.870360] Key type .llcrypt registered [ 3304.939735] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3305.360647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3305.373597] alg: No test for adler32 (adler32-zlib) [ 3306.294743] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3306.420863] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3308.032180] Key type lgssc registered [ 3308.687723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3312.507785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3312.519352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3313.795429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.610984] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3319.423607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.655929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.502907] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3323.837384] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3325.641833] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3329.024211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.233782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3331.793601] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3334.232360] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3339.503256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 3349.990439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.992901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3349.996386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3349.999506] Lustre: Skipped 1 previous similar message [ 3350.004303] LustreError: Skipped 1 previous similar message [ 3351.343613] Lustre: server umount lustre-OST0000 complete [ 3355.107594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.109467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.111750] Lustre: Skipped 1 previous similar message [ 3355.118329] Lustre: Skipped 2 previous similar messages [ 3359.358408] Lustre: server umount lustre-MDT0000 complete [ 3360.227027] LustreError: 195133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.234928] LustreError: 195133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3361.109154] LustreError: 196755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536568 with bad export cookie 5294510571019968491 [ 3361.116606] LustreError: 196755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3361.117236] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3361.271230] Lustre: server umount lustre-MDT0001 complete [ 3365.433903] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3366.986076] Key type lgssc unregistered [ 3367.112332] LNet: 198463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.114498] LNetError: 198463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.123449] LNet: Removed LNI 192.168.204.102@tcp [ 3367.547889] Key type .llcrypt unregistered [ 3367.549968] Key type ._llcrypt unregistered [ 3377.212683] Key type ._llcrypt registered [ 3377.215442] Key type .llcrypt registered [ 3377.290193] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3377.891132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3377.897811] alg: No test for adler32 (adler32-zlib) [ 3378.813180] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3378.959172] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3380.560183] Key type lgssc registered [ 3381.198353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3385.238130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3385.249741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.466263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3388.269302] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3392.457975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.652646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3394.554935] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3397.173755] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3398.825772] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3401.644758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3401.807605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.075536] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3406.127475] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3411.054112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 3416.551327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.552532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3416.553314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3416.559637] Lustre: Skipped 1 previous similar message [ 3420.878126] Lustre: server umount lustre-OST0000 complete [ 3423.713126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3423.717305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.724233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.727104] Lustre: Skipped 1 previous similar message [ 3426.786076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3426.787384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.604570] Lustre: server umount lustre-MDT0000 complete [ 3430.319183] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536637 with bad export cookie 5301447279538533984 [ 3430.326242] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.327762] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3430.486508] Lustre: server umount lustre-MDT0001 complete [ 3435.175273] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3436.610277] Key type lgssc unregistered [ 3436.784622] LNet: 203020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3436.790094] LNetError: 203020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3436.799565] LNet: Removed LNI 192.168.204.102@tcp [ 3437.237990] Key type .llcrypt unregistered [ 3437.240326] Key type ._llcrypt unregistered [ 3446.810984] Key type ._llcrypt registered [ 3446.812750] Key type .llcrypt registered [ 3446.878979] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3447.311671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3447.318434] alg: No test for adler32 (adler32-zlib) [ 3448.210408] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3448.324958] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3449.936173] Key type lgssc registered [ 3450.597578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.586931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.596205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.794356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3457.567287] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3461.557032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.703174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3463.281724] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3465.862344] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3467.534107] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3470.752856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3470.981190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.727925] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3476.158273] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.586696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 3490.134182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3490.159496] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3490.253644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3492.885495] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3495.368356] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3496.182424] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3506.985841] Lustre: Permanently deactivating lustre-OST0001 [ 3506.997139] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3515.401929] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3519.701610] Lustre: Permanently reactivating lustre-OST0001 [ 3519.711901] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 3519.716283] Lustre: Skipped 2 previous similar messages [ 3523.780959] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.789187] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3523.791701] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3523.795645] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3523.812492] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3532.264231] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.268142] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3532.269543] Lustre: Skipped 2 previous similar messages [ 3536.463692] Lustre: server umount lustre-OST0001 complete [ 3537.382229] LustreError: 205843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3537.388602] LustreError: 205843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3542.501894] LustreError: 205843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3542.503473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.506352] LustreError: 205843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3542.506584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3542.506598] Lustre: Skipped 1 previous similar message [ 3542.513564] Lustre: Skipped 1 previous similar message [ 3544.280383] Lustre: server umount lustre-OST0000 complete [ 3547.620580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.622769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.628401] Lustre: Skipped 1 previous similar message [ 3547.632395] Lustre: Skipped 1 previous similar message [ 3552.128832] Lustre: server umount lustre-MDT0000 complete [ 3552.738267] LustreError: 204248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.748311] LustreError: 204248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3553.963315] LustreError: 204229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536761 with bad export cookie 1732684662076217480 [ 3553.970426] LustreError: 204229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3553.972839] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.147218] Lustre: server umount lustre-MDT0001 complete [ 3558.771963] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3560.152839] Key type lgssc unregistered [ 3560.313562] LNet: 208865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3560.318808] LNetError: 208865:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3560.329770] LNet: Removed LNI 192.168.204.102@tcp [ 3560.808394] Key type .llcrypt unregistered [ 3560.810860] Key type ._llcrypt unregistered [ 3580.219701] Key type ._llcrypt registered [ 3580.221474] Key type .llcrypt registered [ 3580.277678] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3580.704797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.781343] alg: No test for adler32 (adler32-zlib) [ 3581.637726] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3581.714866] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3583.288186] Key type lgssc registered [ 3583.868197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3587.671650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3587.685070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3588.819069] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3588.833812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3588.934773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.874507] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3595.052241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.118543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3595.139636] Lustre: 211029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3595.319511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3597.105186] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3599.756199] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3601.337323] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3603.801050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.895203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3603.959530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.941727] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3606.240936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 3608.320459] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3610.099886] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 3618.986923] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3630.113983] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 3636.890565] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 3645.667365] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3666.896850] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 3666.901334] Lustre: Skipped 2 previous similar messages [ 3704.910646] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 3704.916337] Lustre: Skipped 4 previous similar messages [ 3774.772882] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 3774.778246] Lustre: Skipped 8 previous similar messages [ 3780.577931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.581418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.581725] LustreError: Skipped 1 previous similar message [ 3780.582672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.586799] Lustre: Skipped 1 previous similar message [ 3782.365386] Lustre: server umount lustre-OST0000 complete [ 3784.673364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3784.678164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.686330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.690051] Lustre: Skipped 1 previous similar message [ 3785.697735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.698939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.195732] Lustre: server umount lustre-MDT0000 complete [ 3791.927988] LustreError: 211022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780536999 with bad export cookie 401285413299052793 [ 3791.935183] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3791.935548] LustreError: 211022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3792.094127] Lustre: server umount lustre-MDT0001 complete [ 3796.919391] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3798.350054] Key type lgssc unregistered [ 3798.516671] LNet: 215489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3798.521745] LNetError: 215489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3798.537575] LNet: Removed LNI 192.168.204.102@tcp [ 3799.005473] Key type .llcrypt unregistered [ 3799.007816] Key type ._llcrypt unregistered [ 3810.801767] Key type ._llcrypt registered [ 3810.803585] Key type .llcrypt registered [ 3810.852828] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3811.341891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3811.367071] alg: No test for adler32 (adler32-zlib) [ 3812.282147] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3812.406177] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3814.016140] Key type lgssc registered [ 3814.619003] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3818.428130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3818.439339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.633596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.277270] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3824.855193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.998048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.428654] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3828.811117] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3830.498461] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3833.594647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.792109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.352114] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_default_debug -1 all [ 3838.992647] Lustre: DEBUG MARKER: oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3844.011607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 3850.209244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.214083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.222316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.284688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3853.286635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3853.297307] Lustre: Skipped 1 previous similar message [ 3856.041944] Lustre: server umount lustre-OST0000 complete [ 3858.403543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.404816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3858.412257] Lustre: Skipped 1 previous similar message [ 3863.522082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.526539] Lustre: Skipped 2 previous similar messages [ 3868.642139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3868.646123] Lustre: Skipped 1 previous similar message [ 3872.224151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3872.308112] Lustre: server umount lustre-MDT0000 complete [ 3873.761267] LustreError: 216864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.771718] LustreError: 216864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3874.057256] LustreError: 218479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780537081 with bad export cookie 17129653618725977129 [ 3874.064798] LustreError: MGC192.168.204.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3874.065146] LustreError: 218479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3874.215621] Lustre: server umount lustre-MDT0001 complete [ 3878.820651] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 3880.376768] Key type lgssc unregistered [ 3880.558832] LNet: 220199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3880.564306] LNetError: 220199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3880.573783] LNet: Removed LNI 192.168.204.102@tcp [ 3881.058677] Key type .llcrypt unregistered [ 3881.061095] Key type ._llcrypt unregistered [ 3891.358843] Key type ._llcrypt registered [ 3891.361418] Key type .llcrypt registered [ 3891.424336] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 3891.738331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3891.744822] alg: No test for adler32 (adler32-zlib) [ 3892.593244] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 3892.673773] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 3894.256161] Key type lgssc registered [ 3894.839342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4158.447123] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4159.932878] Key type lgssc unregistered [ 4160.110727] LNet: 222164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4160.115674] LNetError: 222164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4160.126617] LNet: Removed LNI 192.168.204.102@tcp [ 4160.585569] Key type .llcrypt unregistered [ 4160.587834] Key type ._llcrypt unregistered [ 4171.992949] Key type ._llcrypt registered [ 4171.994366] Key type .llcrypt registered [ 4172.053962] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing set_hostid [ 4180.180610] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing load_modules_local [ 4180.603840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4180.651878] alg: No test for adler32 (adler32-zlib) [ 4181.522302] Lustre: Lustre: Build Version: 2.17.53_57_g5196e64 [ 4181.619594] LNet: Added LNI 192.168.204.102@tcp [8/256/0/180] [ 4183.208134] Key type lgssc registered [ 4183.842207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4198.784063] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4063 sec ========== 21:43:25 (1780537405) [ 4199.618243] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:43:26 (1780537406) === [ 4201.130235] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:43:27 (1780537407) === [ 4219.671390] Lustre: DEBUG MARKER: oleg402-server.virtnet: executing unload_modules_local [ 4221.243256] Key type lgssc unregistered [ 4221.422748] LNet: 227324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4221.427561] LNetError: 227324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4221.438652] LNet: Removed LNI 192.168.204.102@tcp [ 4221.903615] Key type .llcrypt unregistered [ 4221.905889] Key type ._llcrypt unregistered