[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511142008 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2432MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009139] LSM: Security Framework initializing [ 0.010054] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.013046] *** VALIDATE selinux *** [ 0.022017] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027134] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029087] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] *** VALIDATE tmpfs *** [ 0.033467] *** VALIDATE proc *** [ 0.034233] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040029] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044203] debug: unmapping init [mem 0xffffffffb0c59000-0xffffffffb0c60fff] [ 0.047141] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056293] rcu: Hierarchical SRCU implementation. [ 0.058438] smp: Bringing up secondary CPUs ... [ 0.059621] x86: Booting SMP configuration: [ 0.060025] .... node #0, CPUs: #1 #2 #3 [ 0.063272] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149120] node 0 deferred pages initialised in 80ms [ 0.151430] devtmpfs: initialized [ 0.153568] x86/mm: Memory block size: 128MB [ 0.157778] gcov: version magic: 0x41383552 [ 0.161380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165175] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168446] pinctrl core: initialized pinctrl subsystem [ 0.171336] [ 0.171901] ************************************************************* [ 0.174018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176018] ** ** [ 0.179018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181017] ** ** [ 0.183018] ** This means that this kernel is built to expose internal ** [ 0.186020] ** IOMMU data structures, which may compromise security on ** [ 0.188016] ** your system. ** [ 0.190017] ** ** [ 0.193019] ** If you see this message and you are not debugging the ** [ 0.195020] ** kernel, report this immediately to your vendor! ** [ 0.197018] ** ** [ 0.199018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202019] ************************************************************* [ 0.205000] NET: Registered protocol family 16 [ 0.207482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216098] cpuidle: using governor menu [ 0.218397] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219511] PCI: Using configuration type 1 for base access [ 0.221124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.230022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232193] cryptd: max_cpu_qlen set to 1000 [ 0.237334] ACPI: Added _OSI(Module Device) [ 0.239073] ACPI: Added _OSI(Processor Device) [ 0.241015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248281] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254382] ACPI: Interpreter enabled [ 0.256061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258017] ACPI: Using IOAPIC for interrupt routing [ 0.260112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.275668] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.278064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.281031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290157] acpiphp: Slot [2] registered [ 0.292204] acpiphp: Slot [5] registered [ 0.293176] acpiphp: Slot [6] registered [ 0.295182] acpiphp: Slot [7] registered [ 0.298014] acpiphp: Slot [8] registered [ 0.301263] acpiphp: Slot [9] registered [ 0.302099] acpiphp: Slot [10] registered [ 0.304137] acpiphp: Slot [3] registered [ 0.305095] acpiphp: Slot [4] registered [ 0.306072] acpiphp: Slot [11] registered [ 0.307096] acpiphp: Slot [12] registered [ 0.308059] acpiphp: Slot [13] registered [ 0.308993] acpiphp: Slot [14] registered [ 0.310082] acpiphp: Slot [15] registered [ 0.312121] acpiphp: Slot [16] registered [ 0.313136] acpiphp: Slot [17] registered [ 0.315102] acpiphp: Slot [18] registered [ 0.316062] acpiphp: Slot [19] registered [ 0.317061] acpiphp: Slot [20] registered [ 0.318019] acpiphp: Slot [21] registered [ 0.318985] acpiphp: Slot [22] registered [ 0.319083] acpiphp: Slot [23] registered [ 0.321083] acpiphp: Slot [24] registered [ 0.322067] acpiphp: Slot [25] registered [ 0.323082] acpiphp: Slot [26] registered [ 0.324063] acpiphp: Slot [27] registered [ 0.325021] acpiphp: Slot [28] registered [ 0.325994] acpiphp: Slot [29] registered [ 0.326094] acpiphp: Slot [30] registered [ 0.327166] acpiphp: Slot [31] registered [ 0.329070] PCI host bridge to bus 0000:00 [ 0.331021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345292] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349324] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352479] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362906] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368957] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377610] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381120] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385870] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.401835] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412011] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.416016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448477] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491796] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.510017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.530013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.540507] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.547017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.554015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.566014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577395] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.601015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.608692] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.616028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645514] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.648293] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.650307] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.651299] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.654355] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.658203] iommu: Default domain type: Passthrough [ 0.659747] SCSI subsystem initialized [ 0.661085] ACPI: bus type USB registered [ 0.661993] usbcore: registered new interface driver usbfs [ 0.663077] usbcore: registered new interface driver hub [ 0.664088] usbcore: registered new device driver usb [ 0.665168] pps_core: LinuxPPS API ver. 1 registered [ 0.666013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.668095] PTP clock support registered [ 0.670062] EDAC MC: Ver: 3.0.0 [ 0.672135] PCI: Using ACPI for IRQ routing [ 0.674942] NetLabel: Initializing [ 0.676013] NetLabel: domain hash size = 128 [ 0.677012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.680088] NetLabel: unlabeled traffic allowed by default [ 0.682539] vgaarb: loaded [ 0.683250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.685012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.690379] clocksource: Switched to clocksource kvm-clock [ 0.797227] VFS: Disk quotas dquot_6.6.0 [ 0.799171] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801825] *** VALIDATE ramfs *** [ 0.802804] *** VALIDATE hugetlbfs *** [ 0.804377] pnp: PnP ACPI init [ 0.808024] pnp: PnP ACPI: found 6 devices [ 0.825708] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.829414] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.831809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.834193] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.837092] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.839740] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.842877] NET: Registered protocol family 2 [ 0.845521] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.850954] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.854734] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.859861] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864312] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867701] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.871427] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.874685] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.878177] NET: Registered protocol family 1 [ 0.881126] RPC: Registered named UNIX socket transport module. [ 0.883460] RPC: Registered udp transport module. [ 0.885362] RPC: Registered tcp transport module. [ 0.887421] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.890198] NET: Registered protocol family 44 [ 0.892174] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.894695] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.897361] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900132] PCI: CLS 0 bytes, default 64 [ 0.902240] Unpacking initramfs... [ 3.768677] debug: unmapping init [mem 0xffff8b67bcc54000-0xffff8b67bffbffff] [ 3.776361] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.780376] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 3.785556] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.940649] Initialise system trusted keyrings [ 5.951772] Key type blacklist registered [ 5.986186] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.012801] zbud: loaded [ 6.034888] *** VALIDATE nfs *** [ 6.039763] *** VALIDATE nfs4 *** [ 6.053976] pstore: using deflate compression [ 6.072339] Platform Keyring initialized [ 6.445527] NET: Registered protocol family 38 [ 6.447393] Key type asymmetric registered [ 6.448797] Asymmetric key parser 'x509' registered [ 6.463853] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.475684] io scheduler mq-deadline registered [ 6.477464] io scheduler kyber registered [ 6.482575] io scheduler bfq registered [ 6.494741] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.502826] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.512626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.522608] ACPI: Power Button [PWRF] [ 6.538257] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.556927] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.584771] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.599804] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.651708] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.693766] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.728692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.740200] Non-volatile memory driver v1.3 [ 6.743510] Linux agpgart interface v0.103 [ 6.872593] virtio_blk virtio1: [vda] 146864 512-byte logical blocks (75.2 MB/71.7 MiB) [ 6.888113] vda: detected capacity change from 0 to 75194368 [ 6.916297] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.921491] vdb: detected capacity change from 0 to 1073741824 [ 6.968665] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.978349] vdc: detected capacity change from 0 to 2621440000 [ 7.007268] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.009637] vdd: detected capacity change from 0 to 2621440000 [ 7.034722] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.037424] vde: detected capacity change from 0 to 4294967296 [ 7.068330] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.071132] vdf: detected capacity change from 0 to 4294967296 [ 7.087094] libphy: Fixed MDIO Bus: probed [ 7.103563] usbcore: registered new interface driver usbserial_generic [ 7.109355] usbserial: USB Serial support registered for generic [ 7.116613] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.127321] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.132474] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.142795] mousedev: PS/2 mouse device common for all mice [ 7.149665] rtc_cmos 00:05: RTC can wake from S4 [ 7.153795] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.156789] rtc_cmos 00:05: registered as rtc0 [ 7.162519] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.169061] intel_pstate: CPU model not supported [ 7.177375] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.181411] hid: raw HID events driver (C) Jiri Kosina [ 7.185053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.189220] usbcore: registered new interface driver usbhid [ 7.194793] usbhid: USB HID core driver [ 7.196677] drop_monitor: Initializing network drop monitor service [ 7.199686] Initializing XFRM netlink socket [ 7.202478] NET: Registered protocol family 10 [ 7.206579] Segment Routing with IPv6 [ 7.208226] NET: Registered protocol family 17 [ 7.210902] mpls_gso: MPLS GSO support [ 7.217594] RAS: Correctable Errors collector initialized. [ 7.220156] AVX version of gcm_enc/dec engaged. [ 7.222138] AES CTR mode by8 optimization enabled [ 7.310470] sched_clock: Marking stable (7310450604, 0)->(8197002924, -886552320) [ 7.320335] registered taskstats version 1 [ 7.323590] Loading compiled-in X.509 certificates [ 7.328697] zswap: loaded using pool lzo/zbud [ 7.432553] Key type big_key registered [ 7.490287] Key type encrypted registered [ 7.493648] ima: No TPM chip found, activating TPM-bypass! [ 7.500752] ima: Allocated hash algorithm: sha1 [ 7.502597] ima: No architecture policies found [ 7.505906] evm: Initialising EVM extended attributes: [ 7.508682] evm: security.selinux [ 7.510735] evm: security.ima [ 7.512614] evm: security.capability [ 7.514993] evm: HMAC attrs: 0x1 [ 7.518770] rtc_cmos 00:05: setting system clock to 2026-08-27 16:09:36 UTC (1787846976) [ 7.532398] debug: unmapping init [mem 0xffffffffb1c03000-0xffffffffb1dfffff] [ 7.540704] debug: unmapping init [mem 0xffffffffb0982000-0xffffffffb0c58fff] [ 7.555383] Write protecting the kernel read-only data: 28672k [ 7.568653] debug: unmapping init [mem 0xffffffffaf003000-0xffffffffaf1fffff] [ 7.587696] debug: unmapping init [mem 0xffffffffaf914000-0xffffffffaf9fffff] [ 7.717978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.736655] systemd[1]: Detected virtualization kvm. [ 7.738534] systemd[1]: Detected architecture x86-64. [ 7.745468] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.829589] systemd[1]: No hostname configured. [ 7.832656] systemd[1]: Set hostname to . [ 7.842109] random: systemd: uninitialized urandom read (16 bytes read) [ 7.846237] systemd[1]: Initializing machine ID from random generator. [ 7.998765] random: ln: uninitialized urandom read (6 bytes read) [ 8.357747] random: systemd: uninitialized urandom read (16 bytes read) [ 8.361805] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 8.399876] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 8.415173] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.331664] device-mapper: uevent: version 1.0.3 [ 10.348380] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.575669] virtio_net virtio0 ens2: renamed from eth0 [ 12.739689] random: fast init done [ 12.896411] scsi host0: ata_piix [ 12.913242] scsi host1: ata_piix [ 12.923058] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.927288] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.038603] random: crng init done [ 18.040612] random: 7 urandom warning(s) missed due to ratelimiting [ 19.478243] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.263154] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.862355] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.807199] SELinux: Disabled at runtime. [ 26.972157] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.998907] systemd[1]: Detected virtualization kvm. [ 27.006656] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.783115] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.791903] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.807363] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.817476] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.823975] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.847634] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.887961] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ 29.553741] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 30.921946] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.029504] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.229156] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.752997] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.987800] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 38.932505] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 40.031326] NFS: Registering the id_resolver key type [ 40.048728] Key type id_resolver registered [ 40.052928] Key type id_legacy registered [ *] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg248-server login: [ 87.658363] hrtimer: interrupt took 5610577 ns [ 95.546787] libcfs: loading out-of-tree module taints kernel. [ 95.697697] Key type ._llcrypt registered [ 95.703493] Key type .llcrypt registered [ 95.837837] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 117.783571] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 120.005862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.026376] alg: No test for adler32 (adler32-zlib) [ 121.881275] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 123.079152] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 124.985839] Key type lgssc registered [ 127.118908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 149.873796] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 195.393759] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 212.772041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.828357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.174174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.206423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.294135] Lustre: lustre-MDT0000: new disk, initializing [ 214.340144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.347850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.917787] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.070372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.252868] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.297809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 236.308664] Lustre: Skipped 1 previous similar message [ 236.459499] Lustre: lustre-MDT0001: new disk, initializing [ 236.546716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.572422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.585405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.584537] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.385419] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.483638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.781644] Lustre: lustre-OST0000: new disk, initializing [ 256.784850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.797973] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.885920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.166541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 262.172824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 262.286125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 262.761196] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 277.904911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 278.070788] Lustre: lustre-OST0001: new disk, initializing [ 278.074912] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 278.080218] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 278.146266] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 284.920622] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.765365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 286.772958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 286.831493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.017556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.769359] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 311.156690] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing check_logdir /tmp/testlogs/ [ 318.093925] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing yml_node [ 324.270240] Lustre: DEBUG MARKER: Client: 2.17.57.87 [ 327.413424] Lustre: DEBUG MARKER: MDS: 2.17.57.87 [ 330.675475] Lustre: DEBUG MARKER: OSS: 2.17.57.87 [ 332.617774] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:15:00 EDT 2026 [ 351.709035] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 353.737785] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 373.740050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.752060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 373.758514] Lustre: Skipped 1 previous similar message [ 376.150177] Lustre: server umount lustre-MDT0000 complete [ 379.399509] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847348 with bad export cookie 11109688428866606102 [ 379.402310] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 379.410399] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 383.981335] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.983147] LustreError: 6528:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.983327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 383.983332] Lustre: Skipped 3 previous similar messages [ 383.993245] Lustre: Skipped 3 previous similar messages [ 384.032215] LustreError: 6528:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 384.994538] LustreError: 6528:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 384.995134] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 385.023283] Lustre: Skipped 2 previous similar messages [ 385.787643] Lustre: server umount lustre-MDT0001 complete [ 390.304894] Lustre: server umount lustre-OST0000 complete [ 394.917834] Lustre: server umount lustre-OST0001 complete [ 401.920766] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 412.627341] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 452.363563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 452.708496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 452.742767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 452.836539] Lustre: lustre-MDT0000: new disk, initializing [ 452.937421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.965909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.267272] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 469.281365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.398692] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 469.440645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 469.443344] Lustre: Skipped 1 previous similar message [ 469.523554] Lustre: lustre-MDT0001: new disk, initializing [ 469.657419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 469.719756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 469.742287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 474.078081] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 483.636798] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 489.348720] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.386341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.641930] Lustre: lustre-OST0000: new disk, initializing [ 496.645519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.661754] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.794739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.025698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 498.039758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.127915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.072557] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 513.928757] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 518.876853] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 519.032262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 523.702258] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 523.931506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 528.874980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.892112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.919729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.328401] Lustre: server umount lustre-OST0000 complete [ 539.116172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.119043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.127066] Lustre: Skipped 2 previous similar messages [ 539.142912] Lustre: Skipped 2 previous similar messages [ 542.915537] Lustre: server umount lustre-MDT0000 complete [ 544.225733] LustreError: 15098:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.239778] LustreError: 15098:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 546.906148] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847515 with bad export cookie 11109688428866607635 [ 546.922238] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.222351] Lustre: server umount lustre-MDT0001 complete [ 550.650194] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:18:38 (1787847518) [ 557.139917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.509816] LustreError: 19319:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.523321] LustreError: 19319:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 557.585696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 561.932583] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 562.658439] LustreError: 19320:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.607590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.581467] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 582.903626] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 589.519796] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.979401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 604.261737] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 612.596854] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.195250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.206072] LustreError: Skipped 1 previous similar message [ 628.208211] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.215785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.244622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 631.907521] Lustre: server umount lustre-OST0000 complete [ 640.482802] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.515446] Lustre: Skipped 2 previous similar messages [ 645.602112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 645.611520] Lustre: Skipped 5 previous similar messages [ 651.233251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 651.415817] Lustre: server umount lustre-MDT0000 complete [ 655.847657] LustreError: 19314:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 655.878016] LustreError: 19314:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 656.122939] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847625 with bad export cookie 11109688428866608517 [ 656.135022] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 656.161509] LustreError: 19299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 657.072854] Lustre: server umount lustre-MDT0001 complete [ 668.077348] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 670.261915] Key type lgssc unregistered [ 670.486674] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.494815] LNetError: 22871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.512587] LNet: Removed LNI 192.168.202.148@tcp [ 671.513958] Key type .llcrypt unregistered [ 671.517960] Key type ._llcrypt unregistered [ 697.100536] Key type ._llcrypt registered [ 697.104059] Key type .llcrypt registered [ 697.197214] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 698.539835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 698.618320] alg: No test for adler32 (adler32-zlib) [ 699.927937] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 700.288671] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 702.024238] Key type lgssc registered [ 703.225812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 710.699313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.724687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.119229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.221220] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 722.416164] LustreError: 24285:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.428920] LustreError: 24285:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 727.521828] LustreError: 24284:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.646656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.921796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 732.687201] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 741.919767] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 748.742771] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 756.248524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.646091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.687886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 764.250521] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 773.098528] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 788.450947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.456187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.480647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 790.218762] Lustre: server umount lustre-OST0000 complete [ 794.593123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 794.607766] LustreError: Skipped 1 previous similar message [ 794.615615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.630302] Lustre: Skipped 1 previous similar message [ 794.642640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.647706] Lustre: Skipped 1 previous similar message [ 798.690655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.693680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.855707] Lustre: server umount lustre-MDT0000 complete [ 803.811932] LustreError: 24284:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 804.991558] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847773 with bad export cookie 3615468341617423167 [ 805.005783] LustreError: 24267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 805.006883] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.605640] Lustre: server umount lustre-MDT0001 complete [ 817.413534] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 820.511557] Key type lgssc unregistered [ 820.800992] LNet: 28262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.803563] LNetError: 28262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.815961] LNet: Removed LNI 192.168.202.148@tcp [ 821.616765] Key type .llcrypt unregistered [ 821.621234] Key type ._llcrypt unregistered [ 849.318660] Key type ._llcrypt registered [ 849.321599] Key type .llcrypt registered [ 849.419435] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 850.538681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.588203] alg: No test for adler32 (adler32-zlib) [ 851.594842] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 851.791406] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 853.416133] Key type lgssc registered [ 854.875169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.613334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.646793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 864.269949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 869.150247] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 874.472195] LustreError: 29676:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 874.486733] LustreError: 29676:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 879.430983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.602383] LustreError: 29677:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 880.030317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 884.871937] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 894.531980] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 901.549479] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 916.775911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 917.099594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.705858] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 933.168850] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 934.539748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 945.126060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 945.128462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 945.152909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 945.161546] Lustre: Skipped 1 previous similar message [ 950.243621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.367982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.381099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 955.381489] Lustre: Skipped 2 previous similar messages [ 955.702200] Lustre: server umount lustre-OST0000 complete [ 962.019329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 962.024846] LustreError: Skipped 1 previous similar message [ 962.030075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.052154] Lustre: Skipped 1 previous similar message [ 962.063478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 965.614161] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.721860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 970.732133] Lustre: Skipped 3 previous similar messages [ 975.328673] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 975.740311] Lustre: server umount lustre-MDT0000 complete [ 975.843043] LustreError: 29676:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.686074] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847949 with bad export cookie 5647505258967489362 [ 980.692913] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.699137] LustreError: 29656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 981.251090] Lustre: server umount lustre-MDT0001 complete [ 995.207452] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 998.358533] Key type lgssc unregistered [ 998.741364] LNet: 33569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.749275] LNetError: 33569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.768447] LNet: Removed LNI 192.168.202.148@tcp [ 999.826785] Key type .llcrypt unregistered [ 999.831481] Key type ._llcrypt unregistered [ 1027.945653] Key type ._llcrypt registered [ 1027.947898] Key type .llcrypt registered [ 1028.046760] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1029.447257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.514464] alg: No test for adler32 (adler32-zlib) [ 1030.770304] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1031.093450] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1032.832210] Key type lgssc registered [ 1034.407420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.033194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.062790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1046.650566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.208634] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1056.231457] LustreError: 34983:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.259072] LustreError: 34983:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1061.347100] LustreError: 34984:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1062.974247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.481201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.686773] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1078.385661] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1085.680641] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1094.417475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.693704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.165502] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1110.860979] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.213150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1129.443253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.452431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.472083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.490486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.491579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1131.539481] Lustre: Skipped 1 previous similar message [ 1133.946918] Lustre: server umount lustre-OST0000 complete [ 1140.209161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1140.227975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.267030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.927851] Lustre: server umount lustre-MDT0000 complete [ 1146.857251] LustreError: 34984:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.881270] LustreError: 34984:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1149.842887] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848118 with bad export cookie 9999985151149129226 [ 1149.844475] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1149.868461] LustreError: 34964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.414913] Lustre: server umount lustre-MDT0001 complete [ 1167.144638] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1170.799944] Key type lgssc unregistered [ 1171.170956] LNet: 38545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.185751] LNetError: 38545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.214177] LNet: Removed LNI 192.168.202.148@tcp [ 1172.211448] Key type .llcrypt unregistered [ 1172.213350] Key type ._llcrypt unregistered [ 1201.194819] Key type ._llcrypt registered [ 1201.196741] Key type .llcrypt registered [ 1201.334129] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1202.584346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1202.862755] alg: No test for adler32 (adler32-zlib) [ 1204.075630] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1204.330237] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1206.043551] Key type lgssc registered [ 1207.658084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.896828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.921035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.483782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.635229] LustreError: 39959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1222.675876] LustreError: 39959:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1223.678458] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1227.760618] LustreError: 39960:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.890935] LustreError: 39959:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.407601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.972019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.945908] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1248.023620] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.232753] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1260.389493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1260.669962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.711384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1267.363287] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1276.448303] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1288.305619] Lustre: lustre-OST0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 1288.677907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1288.695035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.704167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.707843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.714278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.718760] Lustre: Skipped 1 previous similar message [ 1291.502694] Lustre: server umount lustre-OST0000 complete [ 1314.273989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.287499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.287995] Lustre: Skipped 1 previous similar message [ 1314.323259] Lustre: Skipped 2 previous similar messages [ 1319.398833] LustreError: 39954:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.431918] Lustre: server umount lustre-MDT0000 complete [ 1323.373564] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848292 with bad export cookie 6072540200496390640 [ 1323.378565] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.386630] LustreError: 39940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1329.856420] Lustre: server umount lustre-MDT0001 complete [ 1341.885707] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1345.208412] Key type lgssc unregistered [ 1345.541554] LNet: 43671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.564029] LNetError: 43671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.593325] LNet: Removed LNI 192.168.202.148@tcp [ 1346.430391] Key type .llcrypt unregistered [ 1346.432200] Key type ._llcrypt unregistered [ 1374.264733] Key type ._llcrypt registered [ 1374.273179] Key type .llcrypt registered [ 1374.439630] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1375.975031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.234732] alg: No test for adler32 (adler32-zlib) [ 1377.545332] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1377.916251] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1379.712199] Key type lgssc registered [ 1381.643780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1391.634608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1391.678429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.089350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.999703] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1403.368844] LustreError: 45085:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.391172] LustreError: 45085:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1408.486180] LustreError: 45086:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.459471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.949468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.963300] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1427.492260] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1435.199737] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1446.157908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1446.462978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.517783] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1463.881421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1464.340475] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1472.088405] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 1474.018500] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.020941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.052115] Lustre: Skipped 1 previous similar message [ 1474.069365] Lustre: Skipped 2 previous similar messages [ 1477.222140] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 1482.310327] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 1482.317869] Lustre: Skipped 3 previous similar messages [ 1486.304140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1486.648568] Lustre: server umount lustre-MDT0000 complete [ 1487.422904] LustreError: 45080:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.950908] LustreError: 45067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848460 with bad export cookie 2026786667782216196 [ 1491.957507] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.970707] LustreError: 45067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1492.543341] LustreError: 45082:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.557501] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [ 1492.571489] LustreError: 45082:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1492.597079] Lustre: Skipped 3 previous similar messages [ 1494.512935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.552409] Lustre: Skipped 1 previous similar message [ 1496.545357] LustreError: 45080:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1496.571164] LustreError: 45080:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1498.563289] Lustre: server umount lustre-MDT0001 complete [ 1514.614852] Lustre: server umount lustre-OST0000 complete [ 1533.025277] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1536.402187] Key type lgssc unregistered [ 1536.823805] LNet: 48950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.828665] LNetError: 48950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.844808] LNet: Removed LNI 192.168.202.148@tcp [ 1538.409892] Key type .llcrypt unregistered [ 1538.412330] Key type ._llcrypt unregistered [ 1570.413202] Key type ._llcrypt registered [ 1570.414770] Key type .llcrypt registered [ 1570.482184] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1571.576929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1571.931752] alg: No test for adler32 (adler32-zlib) [ 1573.060913] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1573.319965] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1575.032159] Key type lgssc registered [ 1576.208769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1584.034562] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1584.054276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.584759] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1604.704490] LustreError: 50337:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 1631.456217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.465657] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1647.683774] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1816.315417] Lustre: server umount lustre-OST0000 complete [ 1838.518922] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1841.912846] Key type lgssc unregistered [ 1842.342600] LNet: 52021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.349713] LNetError: 52021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.366256] LNet: Removed LNI 192.168.202.148@tcp [ 1843.607224] Key type .llcrypt unregistered [ 1843.612431] Key type ._llcrypt unregistered [ 1873.715308] Key type ._llcrypt registered [ 1873.724084] Key type .llcrypt registered [ 1873.841147] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 1875.007393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1875.054182] alg: No test for adler32 (adler32-zlib) [ 1876.112742] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 1876.325453] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 1878.056262] Key type lgssc registered [ 1879.172008] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.364288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.386970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.801885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.325521] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1897.957851] LustreError: 53440:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.985417] LustreError: 53440:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1900.275553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.638957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.121363] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1913.331141] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.716186] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.472627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.860307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.986959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1934.206426] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 1942.576570] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1948.643834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.649302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.676996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.695758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1950.698847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.027903] Lustre: server umount lustre-OST0000 complete [ 1957.345872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1957.358650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.378422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.384831] Lustre: Skipped 1 previous similar message [ 1960.933529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.935840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1960.957789] Lustre: Skipped 1 previous similar message [ 1963.130639] Lustre: server umount lustre-MDT0000 complete [ 1966.054639] LustreError: 54443:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.077617] LustreError: 54443:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1967.048157] LustreError: 53421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848936 with bad export cookie 12006791480298631435 [ 1967.065809] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.454705] Lustre: server umount lustre-MDT0001 complete [ 1979.817977] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 1982.568956] Key type lgssc unregistered [ 1982.908823] LNet: 56976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.913052] LNetError: 56976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.924464] LNet: Removed LNI 192.168.202.148@tcp [ 1983.944674] Key type .llcrypt unregistered [ 1983.947802] Key type ._llcrypt unregistered [ 2011.102182] Key type ._llcrypt registered [ 2011.103779] Key type .llcrypt registered [ 2011.276675] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2012.350107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.360585] alg: No test for adler32 (adler32-zlib) [ 2013.662786] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2014.030774] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2015.752793] Key type lgssc registered [ 2016.967916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2027.040823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2027.095872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.720360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2047.840402] LustreError: 58379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 2074.528194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.576862] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2092.057709] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2099.784894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.430176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.549893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2105.830564] LustreError: 58404:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.850383] LustreError: 58404:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2106.311610] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2109.935527] LustreError: 58405:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2109.953749] LustreError: 58405:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2115.054019] LustreError: 59415:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2115.085397] LustreError: 59415:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2117.195225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.741689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2123.555407] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2133.883579] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.279271] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2146.793798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2146.800455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.816056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.326926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2148.335749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2148.344069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.389144] Lustre: Skipped 1 previous similar message [ 2151.368599] Lustre: server umount lustre-OST0000 complete [ 2179.052411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2179.056449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.115195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.122719] Lustre: Skipped 1 previous similar message [ 2181.004905] Lustre: server umount lustre-MDT0000 complete [ 2184.165673] LustreError: 61111:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.180105] LustreError: 61111:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2184.979556] LustreError: 58385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849153 with bad export cookie 7003717607277215540 [ 2184.986143] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.999313] LustreError: 58385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2191.636898] Lustre: server umount lustre-MDT0001 complete [ 2205.832794] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2208.759390] Key type lgssc unregistered [ 2209.062637] LNet: 62132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.068190] LNetError: 62132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.087261] LNet: Removed LNI 192.168.202.148@tcp [ 2209.827724] Key type .llcrypt unregistered [ 2209.833674] Key type ._llcrypt unregistered [ 2238.086326] Key type ._llcrypt registered [ 2238.088608] Key type .llcrypt registered [ 2238.200656] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2239.738606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.823223] alg: No test for adler32 (adler32-zlib) [ 2240.888201] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2241.129460] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2242.808366] Key type lgssc registered [ 2244.238605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2253.629949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2253.650308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.417476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.057572] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2265.581028] LustreError: 63550:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.598500] LustreError: 63550:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2269.769898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2270.209755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2275.740433] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2285.159869] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2293.068522] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.266436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.654255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2308.559923] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2316.606264] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2325.253729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2335.723142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2335.725534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.738425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2335.754174] Lustre: Skipped 1 previous similar message [ 2335.784656] Lustre: Skipped 1 previous similar message [ 2340.835906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2340.849273] Lustre: Skipped 1 previous similar message [ 2342.105420] Lustre: server umount lustre-OST0000 complete [ 2346.995249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2347.015980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.069246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.073848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.079869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.103654] Lustre: Skipped 1 previous similar message [ 2352.468866] Lustre: server umount lustre-MDT0000 complete [ 2356.203390] LustreError: 64278:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.219411] LustreError: 64278:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2356.631739] LustreError: 63530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849325 with bad export cookie 11545345811198696840 [ 2356.652580] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.099743] Lustre: server umount lustre-MDT0001 complete [ 2373.912870] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2377.574920] Key type lgssc unregistered [ 2377.927359] LNet: 67110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2377.936646] LNetError: 67110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2377.953626] LNet: Removed LNI 192.168.202.148@tcp [ 2379.021095] Key type .llcrypt unregistered [ 2379.026385] Key type ._llcrypt unregistered [ 2410.568860] Key type ._llcrypt registered [ 2410.571319] Key type .llcrypt registered [ 2410.718714] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2412.063593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.113728] alg: No test for adler32 (adler32-zlib) [ 2413.305814] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2413.609691] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2415.259118] Key type lgssc registered [ 2416.610372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.248984] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:49:56 (1787849396) [ 2435.320186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.349651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.832301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.359456] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2446.318670] LustreError: 68817:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.344219] LustreError: 68817:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2449.928513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.225243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2454.727232] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2463.442455] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2471.003434] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2479.883397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2480.371513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.394191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2488.456958] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2498.167860] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.508483] Lustre: Failing over lustre-MDT0000 [ 2506.111945] Lustre: server umount lustre-MDT0000 complete [ 2506.219910] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.242756] LustreError: 70307:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.264651] LustreError: 70307:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2510.414103] LustreError: 68811:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.527468] LustreError: 68811:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2515.561918] LustreError: 68811:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2517.866249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.010428] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.189800] Lustre: *** cfs_fail_loc=135, val=0*** [ 2518.302061] LustreError: 71905:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2518.309093] LustreError: 71905:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -2 [ 2518.317627] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2518.317627] [ 2518.330343] LustreError: MGC192.168.202.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2518.360507] LustreError: 71893:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2518.365474] LustreError: 71893:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2518.370620] LustreError: 71893:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2518.381671] LustreError: 71893:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2518.423367] Lustre: server umount lustre-MDT0000 complete [ 2518.425950] LustreError: 71893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2520.644931] LustreError: 68813:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.668441] LustreError: 68813:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2528.343047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.508948] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.522239] LustreError: 68808:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2529.018831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.094374] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2530.887833] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2534.390905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2534.464591] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2534.532543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2536.498130] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2549.735563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.737176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.749910] Lustre: Skipped 3 previous similar messages [ 2549.762198] Lustre: Skipped 1 previous similar message [ 2554.851050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.856356] Lustre: Skipped 1 previous similar message [ 2555.526484] Lustre: server umount lustre-OST0000 complete [ 2565.091599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2565.095579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.100842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2565.120097] Lustre: Skipped 1 previous similar message [ 2567.083754] Lustre: server umount lustre-MDT0000 complete [ 2570.211956] LustreError: 68816:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.225953] LustreError: 68816:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 2571.890035] LustreError: 71396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849540 with bad export cookie 14919281348867397827 [ 2571.890790] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.902598] LustreError: 71396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2572.218765] Lustre: server umount lustre-MDT0001 complete [ 2587.716416] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2590.860376] Key type lgssc unregistered [ 2591.179777] LNet: 73692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.186561] LNetError: 73692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.205228] LNet: Removed LNI 192.168.202.148@tcp [ 2592.429570] Key type .llcrypt unregistered [ 2592.433442] Key type ._llcrypt unregistered [ 2620.325171] Key type ._llcrypt registered [ 2620.331331] Key type .llcrypt registered [ 2620.470465] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2621.738947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.771235] alg: No test for adler32 (adler32-zlib) [ 2623.028219] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2623.343525] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2625.072599] Key type lgssc registered [ 2626.340307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2636.710180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2636.724449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2638.147608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.279214] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2648.546735] LustreError: 75108:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.568700] LustreError: 75108:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2651.319075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.700678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2656.827768] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2665.360459] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2672.357479] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2680.748540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2681.287752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2683.331120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2688.361935] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2698.918960] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2706.295372] Lustre: Failing over lustre-MDT0000 [ 2706.876665] Lustre: server umount lustre-MDT0000 complete [ 2706.928718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2706.955447] LustreError: 76728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.155667] LustreError: 75102:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.171810] LustreError: 75102:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2716.225586] LustreError: 76728:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.254582] LustreError: 76728:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2719.220133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.304937] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.429736] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2719.527624] LustreError: 78196:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2719.539537] LustreError: 78196:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -9 [ 2719.559863] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2719.559863] [ 2719.566069] LustreError: MGC192.168.202.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2719.581878] LustreError: 78185:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2719.591586] LustreError: 78185:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2719.613146] LustreError: 78185:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2719.620904] LustreError: 78185:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2719.656639] Lustre: server umount lustre-MDT0000 complete [ 2719.658728] LustreError: 78185:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2721.356756] LustreError: 75104:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.383053] LustreError: 75104:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2727.044674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.136458] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2727.147057] LustreError: 75100:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2727.421715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2727.461439] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2727.540178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2732.053705] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2732.527397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2732.544954] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2732.588826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2742.756826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.760129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.779240] Lustre: Skipped 3 previous similar messages [ 2742.779628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.796356] Lustre: Skipped 1 previous similar message [ 2747.931490] Lustre: server umount lustre-OST0000 complete [ 2752.995928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.022908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.036279] Lustre: Skipped 1 previous similar message [ 2758.115448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.123811] Lustre: Skipped 1 previous similar message [ 2762.210802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.221107] Lustre: Skipped 1 previous similar message [ 2766.305386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.459318] Lustre: server umount lustre-MDT0000 complete [ 2767.339907] LustreError: 75107:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.365752] LustreError: 75107:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 2770.770739] LustreError: 75089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849739 with bad export cookie 16477215731678017191 [ 2770.785757] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.812861] LustreError: 75089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.234353] Lustre: server umount lustre-MDT0001 complete [ 2785.673280] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2788.976515] Key type lgssc unregistered [ 2789.199077] LNet: 79985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.207555] LNetError: 79985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.221895] LNet: Removed LNI 192.168.202.148@tcp [ 2790.163164] Key type .llcrypt unregistered [ 2790.164725] Key type ._llcrypt unregistered [ 2804.530253] Key type ._llcrypt registered [ 2804.532908] Key type .llcrypt registered [ 2804.638818] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2805.007311] Key type .llcrypt unregistered [ 2805.009462] Key type ._llcrypt unregistered [ 2832.166062] Key type ._llcrypt registered [ 2832.168770] Key type .llcrypt registered [ 2832.370406] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2834.105219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2834.379534] alg: No test for adler32 (adler32-zlib) [ 2835.587468] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 2835.928086] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 2837.688159] Key type lgssc registered [ 2839.570294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2847.247262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2847.264693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.662449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.444674] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2858.979694] LustreError: 81893:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.988690] LustreError: 81893:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2863.631708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.217990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2870.488682] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2879.607687] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2886.930920] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2895.923504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.229773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.776189] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 2911.990575] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2918.788037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2934.241461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.254866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.261620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.284177] Lustre: Skipped 1 previous similar message [ 2934.306118] Lustre: Skipped 1 previous similar message [ 2938.595230] Lustre: server umount lustre-OST0000 complete [ 2944.482993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.510801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.867828] Lustre: server umount lustre-MDT0000 complete [ 2949.602498] LustreError: 81887:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.634128] LustreError: 81887:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2953.011254] LustreError: 81873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849921 with bad export cookie 659166395165837013 [ 2953.027687] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.314371] Lustre: server umount lustre-MDT0001 complete [ 2966.465772] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 2969.165522] Key type lgssc unregistered [ 2969.433749] LNet: 85451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2969.460335] LNetError: 85451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2969.501383] LNet: Removed LNI 192.168.202.148@tcp [ 2970.577289] Key type .llcrypt unregistered [ 2970.581359] Key type ._llcrypt unregistered [ 2997.080484] Key type ._llcrypt registered [ 2997.082388] Key type .llcrypt registered [ 2997.243584] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 2998.862915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.890644] alg: No test for adler32 (adler32-zlib) [ 2999.941336] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3000.220560] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3001.888521] Key type lgssc registered [ 3002.944688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3012.312298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3012.323911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.225709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3020.772411] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3024.353958] LustreError: 86866:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.375609] LustreError: 86866:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3029.480190] LustreError: 86867:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.983896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.682661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.453378] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3047.084780] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3056.493272] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3066.964479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3067.654074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.307493] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3085.051185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3085.508407] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3092.449252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.461993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.501271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.521182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3095.523095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3095.534959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3098.402333] Lustre: server umount lustre-OST0000 complete [ 3103.207481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3103.213769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.230993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.243168] Lustre: Skipped 1 previous similar message [ 3105.767401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.768764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3105.799860] Lustre: Skipped 1 previous similar message [ 3109.355883] Lustre: server umount lustre-MDT0000 complete [ 3110.883361] LustreError: 86863:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.899766] LustreError: 86863:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3113.756110] LustreError: 88712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850082 with bad export cookie 6296963273178773471 [ 3113.765089] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.767826] LustreError: 88712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3114.133355] Lustre: server umount lustre-MDT0001 complete [ 3128.050268] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3131.592624] Key type lgssc unregistered [ 3131.838684] LNet: 90418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3131.844317] LNetError: 90418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3131.874833] LNet: Removed LNI 192.168.202.148@tcp [ 3133.064556] Key type .llcrypt unregistered [ 3133.066529] Key type ._llcrypt unregistered [ 3163.289407] Key type ._llcrypt registered [ 3163.293182] Key type .llcrypt registered [ 3163.410113] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3164.781909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3164.910209] alg: No test for adler32 (adler32-zlib) [ 3166.224877] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3166.588981] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3168.344170] Key type lgssc registered [ 3169.679432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3178.957302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3178.976291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.436474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.956816] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3190.765561] LustreError: 91834:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.787341] LustreError: 91834:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3195.878448] LustreError: 91835:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.602116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.988253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.766975] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3210.343819] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3216.989652] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3224.506064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3225.022254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.045568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3231.904354] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3240.258764] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3256.803862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3256.806327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3256.819136] LustreError: Skipped 1 previous similar message [ 3256.849595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.843488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3257.844794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.845708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3262.613872] Lustre: server umount lustre-OST0000 complete [ 3267.066725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.080140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3267.087855] Lustre: Skipped 2 previous similar messages [ 3272.162780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.174498] Lustre: Skipped 1 previous similar message [ 3277.283181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.300443] Lustre: Skipped 1 previous similar message [ 3281.378623] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3281.696694] Lustre: server umount lustre-MDT0000 complete [ 3282.415682] LustreError: 91835:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3282.435767] LustreError: 91835:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3287.523778] LustreError: 91830:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.556951] LustreError: 91830:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3288.222031] LustreError: 91816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850257 with bad export cookie 15272762104514639497 [ 3288.234609] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.252939] LustreError: 91816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3288.964342] Lustre: server umount lustre-MDT0001 complete [ 3301.053595] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3304.713967] Key type lgssc unregistered [ 3304.995614] LNet: 95396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.000044] LNetError: 95396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.017676] LNet: Removed LNI 192.168.202.148@tcp [ 3305.848171] Key type .llcrypt unregistered [ 3305.850872] Key type ._llcrypt unregistered [ 3332.312174] Key type ._llcrypt registered [ 3332.315099] Key type .llcrypt registered [ 3332.415779] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3333.564599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.639235] alg: No test for adler32 (adler32-zlib) [ 3334.735287] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3334.965267] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3336.656177] Key type lgssc registered [ 3337.979846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.931788] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.962556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3362.658219] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3367.776422] LustreError: 96798:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 3394.529474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.090692] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3409.454248] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3416.117492] Lustre: server umount lustre-OST0000 complete [ 3425.818453] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 13:06:33 (1787850393) [ 3434.939194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.510415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.608795] LustreError: 98439:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.626876] LustreError: 98439:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3441.599723] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3445.739505] LustreError: 98438:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.857980] LustreError: 98439:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.310221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.745572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3456.649359] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3465.188532] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3473.412712] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.259350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.713785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.430618] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3498.651339] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3505.362785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3522.020268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3522.033107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.045342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.071566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.072706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3524.118141] Lustre: Skipped 1 previous similar message [ 3528.304053] Lustre: server umount lustre-OST0000 complete [ 3533.793597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3533.808576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.827516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3538.993723] Lustre: server umount lustre-MDT0000 complete [ 3539.426445] LustreError: 98435:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3539.460894] LustreError: 98435:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3543.071608] LustreError: 101404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850512 with bad export cookie 2101480127538881455 [ 3543.073228] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3543.095153] LustreError: 101404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3543.421244] Lustre: server umount lustre-MDT0001 complete [ 3558.246830] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3561.200442] Key type lgssc unregistered [ 3561.521606] LNet: 102159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.533830] LNetError: 102159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.570627] LNet: Removed LNI 192.168.202.148@tcp [ 3562.500786] Key type .llcrypt unregistered [ 3562.502465] Key type ._llcrypt unregistered [ 3612.866759] Key type ._llcrypt registered [ 3612.870190] Key type .llcrypt registered [ 3612.929318] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3614.056793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3614.141356] alg: No test for adler32 (adler32-zlib) [ 3615.316517] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3615.649716] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3617.352190] Key type lgssc registered [ 3618.472426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3627.396617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3627.408118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.875471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.327968] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3638.261633] LustreError: 104349:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3638.300247] LustreError: 104349:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3643.373723] LustreError: 104350:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.312361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.756388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3649.219364] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3658.044353] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.245865] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3672.152287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3672.816268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3673.855402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3680.698477] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3690.367158] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.802942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3704.809135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3704.815962] LustreError: Skipped 1 previous similar message [ 3704.826070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3704.834941] Lustre: Skipped 1 previous similar message [ 3704.851768] Lustre: Skipped 1 previous similar message [ 3706.701978] Lustre: server umount lustre-OST0000 complete [ 3711.460069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3711.472584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.500420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.041204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3715.041543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.581318] Lustre: server umount lustre-MDT0000 complete [ 3720.162431] LustreError: 105370:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.180160] LustreError: 105370:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3721.667485] LustreError: 104328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850690 with bad export cookie 11035304566464486336 [ 3721.676920] LustreError: 104328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3721.680136] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3722.108655] Lustre: server umount lustre-MDT0001 complete [ 3736.050510] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 3739.332964] Key type lgssc unregistered [ 3739.699547] LNet: 107888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3739.715340] LNetError: 107888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3739.736518] LNet: Removed LNI 192.168.202.148@tcp [ 3740.876318] Key type .llcrypt unregistered [ 3740.879843] Key type ._llcrypt unregistered [ 3762.609438] Key type ._llcrypt registered [ 3762.613749] Key type .llcrypt registered [ 3762.803818] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3764.157586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3764.206856] alg: No test for adler32 (adler32-zlib) [ 3765.863879] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 3766.215871] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 3768.048494] Key type lgssc registered [ 3769.846662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3779.963510] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3780.015486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3795.616397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3800.736667] LustreError: 109131:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 3827.427746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3834.357852] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3843.153249] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3851.646783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3852.079790] LustreError: MGC192.168.202.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3852.097553] LustreError: 110156:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3852.117413] LustreError: 110156:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3852.135921] LustreError: 110156:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3852.157616] LustreError: 109137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850821 with bad export cookie 12098260198360715156 [ 3852.184190] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3852.315637] Lustre: server umount lustre-MDT0000 complete [ 3852.321183] LustreError: 110156:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3878.176417] Lustre: server umount lustre-OST0000 complete [ 3891.393647] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 3904.021776] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 3946.282417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.577646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3946.625645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3946.755332] Lustre: lustre-MDT0000: new disk, initializing [ 3946.844816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3946.869282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3952.040985] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3963.930244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.051424] Lustre: 112951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.116138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3964.121812] Lustre: Skipped 1 previous similar message [ 3964.194704] Lustre: lustre-MDT0001: new disk, initializing [ 3964.283828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3964.326943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3964.342432] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3969.222678] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 3981.164637] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3988.248936] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3996.890179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3997.211245] Lustre: lustre-OST0000: new disk, initializing [ 3997.219502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3997.231948] Lustre: 114921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3997.396289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.740584] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4006.550317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4006.576491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4006.722287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4018.137688] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4023.428947] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4023.690924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4029.998276] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4030.288551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4032.486300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4032.497911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.523503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.602578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.609689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.610545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4038.728092] Lustre: server umount lustre-OST0000 complete [ 4042.722120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.746991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4042.749843] Lustre: Skipped 1 previous similar message [ 4047.842516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.849725] Lustre: Skipped 1 previous similar message [ 4048.925886] Lustre: server umount lustre-MDT0000 complete [ 4052.482150] LustreError: 112944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851021 with bad export cookie 12098260198360715261 [ 4052.487113] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4052.493036] LustreError: 112944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4052.729622] Lustre: server umount lustre-MDT0001 complete [ 4060.355907] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:17:08 (1787851028) [ 4063.506411] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4080.111909] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 4092.050773] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4132.622368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.094789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4133.122559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4133.219098] Lustre: lustre-MDT0000: new disk, initializing [ 4133.293545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.313309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4139.990550] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4150.984495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.119445] Lustre: 119820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.169883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4151.173576] Lustre: Skipped 1 previous similar message [ 4151.236711] Lustre: lustre-MDT0001: new disk, initializing [ 4151.319052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.372199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4151.388324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4155.517343] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4165.479502] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4171.163569] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.411193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.686309] Lustre: lustre-OST0000: new disk, initializing [ 4178.689520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4178.706776] Lustre: 121787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4178.825774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.641416] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4187.155950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.167264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4187.262121] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4195.925074] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4200.942535] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4201.177807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4206.438336] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4206.748232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4212.708029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.709121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4212.728100] Lustre: Skipped 1 previous similar message [ 4212.730418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4212.746193] Lustre: Skipped 1 previous similar message [ 4215.080962] Lustre: server umount lustre-OST0000 complete [ 4222.945931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.952494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4222.957342] Lustre: Skipped 1 previous similar message [ 4222.958591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4222.995374] Lustre: Skipped 1 previous similar message [ 4225.737403] Lustre: server umount lustre-MDT0000 complete [ 4230.234448] LustreError: 119812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851199 with bad export cookie 12098260198360716143 [ 4230.251604] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4230.595828] Lustre: server umount lustre-MDT0001 complete [ 4241.229412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.788853] LustreError: 124008:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.812092] LustreError: 124008:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4241.891103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.015205] LustreError: 124009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.033727] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4251.110101] LustreError: 124008:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4256.244138] LustreError: 124009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.101171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.054591] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4273.030691] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4279.194395] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4287.071149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4294.496305] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4302.645659] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4319.556487] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4326.241121] Lustre: server umount lustre-OST0000 complete [ 4332.002528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.022334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.041331] Lustre: Skipped 1 previous similar message [ 4332.050126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4340.901952] Lustre: server umount lustre-MDT0000 complete [ 4342.757482] Lustre: 108468:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851295/real 1787851295] req@00000000d0f922ea x1874697372423040/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787851311 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4342.805493] LustreError: 124009:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.828703] LustreError: 124009:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4343.844783] Lustre: 108468:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851296/real 1787851296] req@00000000a45a639a x1874697372424064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787851312 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4344.916700] LustreError: 123988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851313 with bad export cookie 12098260198360717025 [ 4344.923465] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.928257] LustreError: 123988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4345.255466] Lustre: server umount lustre-MDT0001 complete [ 4360.204966] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 4363.345495] Key type lgssc unregistered [ 4363.659953] LNet: 127666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4363.675605] LNetError: 127666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4363.710461] LNet: Removed LNI 192.168.202.148@tcp [ 4364.716535] Key type .llcrypt unregistered [ 4364.718982] Key type ._llcrypt unregistered [ 4383.860280] Key type ._llcrypt registered [ 4383.862142] Key type .llcrypt registered [ 4383.950915] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 4399.115791] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 4401.091212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4401.167905] alg: No test for adler32 (adler32-zlib) [ 4402.329844] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 4402.568857] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 4404.329417] Key type lgssc registered [ 4405.780477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4449.535975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4449.545880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.798337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4450.849717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4450.992656] Lustre: lustre-MDT0000: new disk, initializing [ 4451.151390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.222266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4457.582770] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4470.122546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.359289] Lustre: 130820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4470.420922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4470.425708] Lustre: Skipped 1 previous similar message [ 4470.517381] Lustre: lustre-MDT0001: new disk, initializing [ 4470.634375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4470.698493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4470.714409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4476.220712] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4486.528555] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4493.104905] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4500.417629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4500.704406] Lustre: lustre-OST0000: new disk, initializing [ 4500.719499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4500.739960] Lustre: 132785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4500.871195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.684748] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4508.769270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4508.790670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4508.922290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4519.508907] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.891722] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4525.191554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4530.442393] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4530.655708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4534.755385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4534.764765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.779565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.786161] Lustre: Skipped 1 previous similar message [ 4534.811970] Lustre: Skipped 1 previous similar message [ 4539.420660] Lustre: server umount lustre-OST0000 complete [ 4545.000320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.014511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.019938] Lustre: Skipped 1 previous similar message [ 4549.691305] LustreError: 134326:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000d6f49e3f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4549.756207] Lustre: server umount lustre-MDT0000 complete [ 4553.595922] LustreError: 130810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851522 with bad export cookie 2277968409144334481 [ 4553.599906] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4553.606981] LustreError: 130810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4554.009109] Lustre: server umount lustre-MDT0001 complete [ 4563.583367] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:25:31 (1787851531) [ 4571.323834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.966816] LustreError: 135198:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4572.000664] LustreError: 135198:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4572.106371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.254068] LustreError: 135199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4577.305791] LustreError: 135199:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4577.780908] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4581.354564] LustreError: 135198:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.466579] LustreError: 135199:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4587.780096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.030540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4593.329496] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4601.291652] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4607.932744] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4612.582202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4612.588274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4612.597062] Lustre: Skipped 2 previous similar messages [ 4612.615110] Lustre: Skipped 1 previous similar message [ 4615.970898] Lustre: server umount lustre-MDT0000 complete [ 4618.722450] LustreError: 135928:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.744271] LustreError: 135928:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4619.716159] LustreError: 135180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851588 with bad export cookie 2277968409144335363 [ 4619.736031] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4620.071931] Lustre: server umount lustre-MDT0001 complete [ 4630.568399] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:26:38 (1787851598) [ 4638.230708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4653.984991] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4659.104714] LustreError: 137616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 4684.832278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.879781] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4700.554635] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4703.019928] Lustre: server umount lustre-OST0000 complete [ 4714.944875] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:28:02 (1787851682) [ 4724.390412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.868472] LustreError: 139062:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4724.938883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.440339] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4741.246560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.686510] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4758.845128] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4767.679656] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4776.208617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.823489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4776.837429] Lustre: Skipped 1 previous similar message [ 4783.235819] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4793.632039] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4804.582757] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4804.583689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4804.603706] Lustre: Skipped 1 previous similar message [ 4804.608907] Lustre: Skipped 2 previous similar messages [ 4809.703621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4809.711866] Lustre: Skipped 4 previous similar messages [ 4810.043850] Lustre: server umount lustre-MDT0000 complete [ 4813.917749] LustreError: 139043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851782 with bad export cookie 2277968409144336028 [ 4813.933416] LustreError: 139043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4813.934964] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4814.344411] Lustre: server umount lustre-MDT0001 complete [ 4829.099934] Lustre: server umount lustre-OST0000 complete [ 4839.995713] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:30:07 (1787851807) [ 4848.261651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.654831] LustreError: 142718:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.676810] LustreError: 142718:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4848.732807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.800269] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4864.783758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4870.493599] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4879.689782] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4887.200281] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4897.850707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.681672] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 4914.701846] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4918.731316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4920.417841] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.692178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4928.373481] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4929.125820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4934.116501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4934.141058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.162792] LustreError: Skipped 1 previous similar message [ 4934.164598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4934.188927] Lustre: Skipped 2 previous similar messages [ 4934.205503] Lustre: Skipped 1 previous similar message [ 4939.268981] LustreError: 144546:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4939.323493] LustreError: 144546:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4939.634140] Lustre: server umount lustre-OST0000 complete [ 4946.913654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4946.917154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.933322] Lustre: server umount lustre-MDT0000 complete [ 4958.481882] LustreError: 142699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851927 with bad export cookie 2277968409144337099 [ 4958.489037] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4958.506877] LustreError: 142699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4958.990396] Lustre: server umount lustre-MDT0001 complete [ 4969.084805] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:32:16 (1787851936) [ 4976.998267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4992.608873] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4997.728651] LustreError: 146885:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 5023.456339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.466961] Lustre: Skipped 2 previous similar messages [ 5030.072438] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5038.778342] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5046.389763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.050533] LustreError: 146910:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.085398] LustreError: 146910:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5047.389200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5051.814870] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5061.056783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.309502] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5074.527251] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5081.566551] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5086.764248] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5086.979197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5091.932449] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5092.131580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5097.443425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.444289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5097.457649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.457659] Lustre: Skipped 3 previous similar messages [ 5097.479338] Lustre: Skipped 2 previous similar messages [ 5100.545942] Lustre: server umount lustre-MDT0000 complete [ 5104.654602] LustreError: 146891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852073 with bad export cookie 2277968409144338121 [ 5104.673923] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.054920] Lustre: server umount lustre-MDT0001 complete [ 5120.493846] Lustre: server umount lustre-OST0000 complete [ 5130.978344] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:34:59 (1787852099) [ 5139.901075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5155.424653] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5160.547108] LustreError: 151079:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 5193.833783] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5203.006432] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5211.086819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.550487] LustreError: 151105:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.572131] LustreError: 151105:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 5211.700792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5216.524544] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5227.083308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.279547] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5246.698529] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5255.289753] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5263.246603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5263.570801] Lustre: lustre-OST0001: new disk, initializing [ 5263.573717] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5263.579615] Lustre: 153955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5270.250056] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5272.648316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5272.656794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5272.780792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5281.696349] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5287.307721] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5287.667839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5293.910389] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5294.626927] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5298.657775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5298.665879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.678242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.678260] Lustre: Skipped 2 previous similar messages [ 5298.692677] Lustre: Skipped 2 previous similar messages [ 5303.305213] Lustre: server umount lustre-OST0000 complete [ 5314.019208] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5314.030055] LustreError: Skipped 1 previous similar message [ 5316.374113] Lustre: server umount lustre-OST0001 complete [ 5327.641305] Lustre: server umount lustre-MDT0000 complete [ 5331.858412] LustreError: 151087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852300 with bad export cookie 2277968409144339073 [ 5331.864946] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.870332] LustreError: 151087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5332.157104] Lustre: server umount lustre-MDT0001 complete [ 5367.663905] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:38:54 (1787852334) [ 5371.373227] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5374.665195] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5377.268527] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:39:04 (1787852344) [ 5385.533213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.773555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5385.836904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5386.155628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.161789] Lustre: Skipped 6 previous similar messages [ 5391.756594] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5401.229293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.408825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5401.460610] Lustre: 157519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5406.690644] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5415.670633] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5423.553657] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5431.833784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5432.133666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5440.366334] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5441.165782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5449.566617] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5454.686627] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5454.879976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5460.622399] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5460.843847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5466.597344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5466.604904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.612969] Lustre: Skipped 4 previous similar messages [ 5466.616791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5466.625842] Lustre: Skipped 5 previous similar messages [ 5469.038292] Lustre: server umount lustre-OST0000 complete [ 5493.926777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5502.678695] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5507.770792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.786531] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5511.860382] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5517.044403] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.073867] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5518.202649] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5518.442924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5523.334152] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5523.616215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5529.341056] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff899247f80800.ost_server_uuid 50 [ 5531.761316] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff899247f80800.ost_server_uuid in FULL state after 0 sec [ 5532.061311] LustreError: 157526:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.48@tcp) returned error from blocking AST (req@000000002860ccf8 x1874698039478144 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000dc61ad22/0x1f9cf832ec9c33db lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.48@tcp remote: 0x1a148391a26acb87 expref: 5 pid: 157526 timeout: 5632 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5532.110769] LustreError: lustre-MDT0000: A client on nid 192.168.202.48@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5532.121096] LustreError: 157514:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.48@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000dc61ad22/0x1f9cf832ec9c33db lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.48@tcp remote: 0x1a148391a26acb87 expref: 6 pid: 157526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5537.776063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5537.791722] LustreError: Skipped 2 previous similar messages [ 5543.990108] Lustre: server umount lustre-OST0000 complete [ 5563.361413] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5563.613116] Lustre: server umount lustre-MDT0000 complete [ 5567.462424] LustreError: 161076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5567.485476] LustreError: 161076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 5567.648898] LustreError: 157509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852536 with bad export cookie 2277968409144340256 [ 5567.658411] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5574.064388] Lustre: server umount lustre-MDT0001 complete [ 5589.931798] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 5593.775976] Key type lgssc unregistered [ 5594.330899] LNet: 163647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5594.354569] LNetError: 163647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5594.380706] LNet: Removed LNI 192.168.202.148@tcp [ 5595.745218] Key type .llcrypt unregistered [ 5595.746689] Key type ._llcrypt unregistered [ 5624.913504] Key type ._llcrypt registered [ 5624.917731] Key type .llcrypt registered [ 5625.050714] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 5626.324608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5626.457885] alg: No test for adler32 (adler32-zlib) [ 5627.620478] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 5627.895205] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 5629.521257] Key type lgssc registered [ 5630.651469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5639.542970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5639.570428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.326893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.639306] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5651.431154] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.454683] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5656.551425] LustreError: 165059:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5656.968747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.356462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5662.592368] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5672.601871] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.175570] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5689.994804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5690.485683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.671937] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5708.089906] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5712.007950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5719.854947] Lustre: Failing over lustre-MDT0000 [ 5720.217235] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 5720.639530] Lustre: server umount lustre-MDT0000 complete [ 5722.597573] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.617552] LustreError: 167146:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5722.627307] Lustre: Skipped 1 previous similar message [ 5722.649099] LustreError: 167146:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5727.716148] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.762283] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5732.838325] LustreError: 165055:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.860348] LustreError: 165055:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5733.796066] LustreError: 165041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.48@tcp arrived at 1787852702 with bad export cookie 463414410132710214 [ 5733.821022] LustreError: 165041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5738.980085] Lustre: 164371:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852691/real 1787852691] req@000000006f47e050 x1874699323733632/t0(0) o400->MGC192.168.202.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852707 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5739.029302] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.171918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.101291] LustreError: 165059:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.122497] LustreError: 165059:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 5749.242622] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x66e60f363c1e48f [ 5749.249118] Lustre: MGC192.168.202.148@tcp: Connection restored to 0@lo (at 0@lo) [ 5749.610740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.688213] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5754.879044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5754.881957] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5755.075577] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5758.126071] Lustre: lustre-MDT0000: Denying connection for new client d73361db-abb2-4c0d-8473-7d0c923f193f (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5763.248779] Lustre: lustre-MDT0000: Denying connection for new client d73361db-abb2-4c0d-8473-7d0c923f193f (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 5769.705439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.719395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.734752] Lustre: Skipped 1 previous similar message [ 5769.749444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.386487] Lustre: server umount lustre-OST0000 complete [ 5780.297871] LustreError: 168794:0:(ldlm_lib.c:3039:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5780.306380] Lustre: 168143:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5780.311435] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5780.326567] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5780.341508] LustreError: 168143:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000018dd0af3 x1874699323754368/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5780.363952] LustreError: 168143:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5780.379464] LustreError: 168143:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5780.394623] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 5780.451982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.473019] Lustre: Skipped 1 previous similar message [ 5780.482846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.493580] Lustre: Skipped 2 previous similar messages [ 5780.766653] Lustre: server umount lustre-MDT0000 complete [ 5785.400764] LustreError: 165040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852754 with bad export cookie 463414410132710543 [ 5785.402994] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.403984] LustreError: 164370:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5785.417387] LustreError: 165040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5785.951474] Lustre: server umount lustre-MDT0001 complete [ 5799.711600] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 5803.832240] Key type lgssc unregistered [ 5804.252962] LNet: 169549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5804.271580] LNetError: 169549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5804.297358] LNet: Removed LNI 192.168.202.148@tcp [ 5805.676679] Key type .llcrypt unregistered [ 5805.680256] Key type ._llcrypt unregistered [ 5833.316570] Key type ._llcrypt registered [ 5833.320552] Key type .llcrypt registered [ 5833.496252] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 5835.690845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5835.719871] alg: No test for adler32 (adler32-zlib) [ 5837.077393] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 5837.489450] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 5839.296210] Key type lgssc registered [ 5841.002432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5853.102377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5853.126814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.904337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.827396] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5864.423194] LustreError: 170966:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.448681] LustreError: 170966:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5869.549365] LustreError: 170967:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.792836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.326601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5879.097633] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5888.266967] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5896.546279] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.373124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5905.708027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.969835] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 5923.423588] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.154179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5934.569336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.570455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5934.591880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5934.615790] Lustre: Skipped 1 previous similar message [ 5937.504650] Lustre: server umount lustre-OST0000 complete [ 5944.801928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.807646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.828443] Lustre: Skipped 2 previous similar messages [ 5947.414717] Lustre: server umount lustre-MDT0000 complete [ 5949.927545] LustreError: 170967:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.979680] LustreError: 170967:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5951.856466] LustreError: 173547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852920 with bad export cookie 10006764961889333225 [ 5951.876534] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5951.883870] LustreError: 173547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5952.259862] Lustre: server umount lustre-MDT0001 complete [ 5965.709912] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 5968.696183] Key type lgssc unregistered [ 5969.037467] LNet: 174503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5969.048476] LNetError: 174503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5969.063723] LNet: Removed LNI 192.168.202.148@tcp [ 5969.782239] Key type .llcrypt unregistered [ 5969.786072] Key type ._llcrypt unregistered [ 6002.446972] Key type ._llcrypt registered [ 6002.449279] Key type .llcrypt registered [ 6002.568456] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6003.629339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6003.787748] alg: No test for adler32 (adler32-zlib) [ 6004.912941] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6005.190024] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6006.824407] Key type lgssc registered [ 6008.175986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6016.491906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6016.516769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.918269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.620288] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6027.239593] LustreError: 176205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.270584] LustreError: 176205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6032.356537] LustreError: 176204:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.566577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.083043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.577470] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6047.879868] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6055.269227] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6064.020501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.593909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.054032] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6082.299905] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6088.126920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6098.423959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.431510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6098.443946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6101.014365] Lustre: server umount lustre-OST0000 complete [ 6108.643633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6108.657360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6108.659540] Lustre: Skipped 2 previous similar messages [ 6108.673386] Lustre: Skipped 2 previous similar messages [ 6112.162116] Lustre: server umount lustre-MDT0000 complete [ 6113.767550] LustreError: 176205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.784897] LustreError: 176205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6116.051530] LustreError: 176186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853085 with bad export cookie 4582068068212980603 [ 6116.068145] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6116.071882] LustreError: 176186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6116.866729] Lustre: server umount lustre-MDT0001 complete [ 6131.083971] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6134.037325] Key type lgssc unregistered [ 6134.391955] LNet: 179744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6134.405801] LNetError: 179744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6134.433583] LNet: Removed LNI 192.168.202.148@tcp [ 6135.475735] Key type .llcrypt unregistered [ 6135.480330] Key type ._llcrypt unregistered [ 6161.150773] Key type ._llcrypt registered [ 6161.157954] Key type .llcrypt registered [ 6161.310779] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6162.952749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6162.992844] alg: No test for adler32 (adler32-zlib) [ 6164.187795] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6164.538088] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6166.273364] Key type lgssc registered [ 6167.485701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6179.694414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6179.708309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.072477] Lustre: *** cfs_fail_loc=135, val=0*** [ 6181.146858] LustreError: 181270:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 6181.152820] LustreError: 181270:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.148@tcp: cfg command failed: rc = -2 [ 6181.161647] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6181.161647] [ 6181.169464] LustreError: MGC192.168.202.148@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6181.183966] LustreError: 181231:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6181.190961] LustreError: 181231:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 6181.201275] LustreError: 181231:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 6181.205425] LustreError: 181231:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6181.261403] Lustre: server umount lustre-MDT0000 complete [ 6181.264818] LustreError: 181231:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6188.257480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6188.613038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.768262] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6198.753720] LustreError: 181642:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.770233] LustreError: 181642:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6203.881711] LustreError: 181643:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.054835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6205.430685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.444534] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6219.399961] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6225.612815] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6230.369937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6230.372383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6230.379980] Lustre: Skipped 1 previous similar message [ 6230.405732] Lustre: Skipped 1 previous similar message [ 6234.768125] Lustre: server umount lustre-MDT0000 complete [ 6235.622663] LustreError: 182372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.640680] LustreError: 182372:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6238.911317] LustreError: 181624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853207 with bad export cookie 11694721031293074765 [ 6238.918148] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6238.921470] LustreError: 181624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6239.635517] Lustre: server umount lustre-MDT0001 complete [ 6251.872661] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6255.158991] Key type lgssc unregistered [ 6255.535914] LNet: 183966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.555415] LNetError: 183966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6255.575940] LNet: Removed LNI 192.168.202.148@tcp [ 6256.524147] Key type .llcrypt unregistered [ 6256.527092] Key type ._llcrypt unregistered [ 6285.215674] Key type ._llcrypt registered [ 6285.220147] Key type .llcrypt registered [ 6285.370373] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6285.890481] Key type .llcrypt unregistered [ 6285.896456] Key type ._llcrypt unregistered [ 6308.368919] Key type ._llcrypt registered [ 6308.371933] Key type .llcrypt registered [ 6308.507612] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6309.520697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6309.585921] alg: No test for adler32 (adler32-zlib) [ 6310.669482] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6310.913418] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6312.568187] Key type lgssc registered [ 6313.703658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6322.423527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6322.442537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6338.016847] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6343.137843] LustreError: 186102:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.148@tcp: failed processing log, type 4: rc = -110 [ 6369.826082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.815722] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6387.331932] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6394.991527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.707079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6395.771659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6400.996825] LustreError: 186128:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.035678] LustreError: 186128:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6401.069836] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6404.069341] LustreError: 187141:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6404.102990] LustreError: 187141:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6409.187413] LustreError: 187140:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.213189] LustreError: 187140:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6411.780548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.333912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.634920] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6426.812588] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6435.082067] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.190959] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6452.196932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6452.209431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.230813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.229323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6453.247412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6453.259902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.437826] Lustre: server umount lustre-OST0000 complete [ 6463.473203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6463.490665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6463.500930] Lustre: Skipped 1 previous similar message [ 6463.501601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6463.522735] Lustre: Skipped 2 previous similar messages [ 6468.581696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6468.591909] Lustre: Skipped 1 previous similar message [ 6473.699347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6473.704598] Lustre: Skipped 1 previous similar message [ 6474.208183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6474.506321] Lustre: server umount lustre-MDT0000 complete [ 6478.821280] LustreError: 187145:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.853035] LustreError: 187145:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6479.311184] LustreError: 186109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853448 with bad export cookie 9166902610807465641 [ 6479.313292] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.321467] LustreError: 186109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6479.919314] Lustre: server umount lustre-MDT0001 complete [ 6497.013540] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6501.803136] Key type lgssc unregistered [ 6502.298622] LNet: 190026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6502.312098] LNetError: 190026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6503.339990] LNet: Removed LNI 192.168.202.148@tcp [ 6505.172726] Key type .llcrypt unregistered [ 6505.176986] Key type ._llcrypt unregistered [ 6536.328412] Key type ._llcrypt registered [ 6536.330794] Key type .llcrypt registered [ 6536.435753] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6537.724262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6537.802441] alg: No test for adler32 (adler32-zlib) [ 6538.932647] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6539.196845] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6540.864330] Key type lgssc registered [ 6542.478898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6551.512299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6551.527511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.871736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.579129] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6558.180554] LustreError: 191458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6558.201356] LustreError: 191458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6562.299633] LustreError: 191459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.353643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.394173] LustreError: 191458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6567.807468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6574.039503] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6584.023333] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.020759] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6598.997528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.477488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6606.968058] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6616.247506] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6622.084126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6627.894901] Lustre: Failing over lustre-MDT0000 [ 6628.471658] Lustre: server umount lustre-MDT0000 complete [ 6628.472862] LustreError: 191454:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.350151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.364371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.424717] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.434843] Lustre: Skipped 1 previous similar message [ 6633.538151] LustreError: 193561:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.569035] LustreError: 193561:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 6642.658504] LustreError: 193561:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.689653] LustreError: 193561:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 6648.800187] Lustre: 190771:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853601/real 1787853601] req@000000005373b38c x1874700280034176/t0(0) o400->MGC192.168.202.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787853617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6648.826119] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6648.940384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6658.024780] LustreError: 190768:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002f710c11 x1874700280039296/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6658.498634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6658.563738] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6659.147524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6663.661066] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6663.664572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6663.697438] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6663.751748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6668.881413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6679.628532] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6691.812202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6691.815424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6691.850293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.389434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.395120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.392666] Lustre: server umount lustre-OST0000 complete [ 6704.610195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6704.611051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.614519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.614528] Lustre: Skipped 1 previous similar message [ 6704.660295] Lustre: Skipped 1 previous similar message [ 6707.795867] Lustre: server umount lustre-MDT0000 complete [ 6709.733431] LustreError: 191459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.763206] LustreError: 191459:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 16 previous similar messages [ 6711.719497] LustreError: 191439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853680 with bad export cookie 11236460569295996082 [ 6711.725605] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.739200] LustreError: 191439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6712.064978] Lustre: server umount lustre-MDT0001 complete [ 6726.001956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6729.313618] Key type lgssc unregistered [ 6729.591747] LNet: 196417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6729.601402] LNetError: 196417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6729.613930] LNet: Removed LNI 192.168.202.148@tcp [ 6730.472472] Key type .llcrypt unregistered [ 6730.478081] Key type ._llcrypt unregistered [ 6757.490189] Key type ._llcrypt registered [ 6757.491476] Key type .llcrypt registered [ 6757.594282] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6758.674514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6758.707062] alg: No test for adler32 (adler32-zlib) [ 6759.820153] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6760.007392] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6761.651085] Key type lgssc registered [ 6763.282871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6771.696638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6771.731626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6773.165174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.591158] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6783.465231] LustreError: 197830:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6783.489207] LustreError: 197830:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6786.624691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6787.167834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6792.227213] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6800.889944] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6807.680955] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.415043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6815.840702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.943048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6823.380497] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6831.189438] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.037326] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6838.638691] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6856.608530] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6863.331271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.357821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.364932] Lustre: Skipped 1 previous similar message [ 6868.449732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6868.455661] Lustre: Skipped 1 previous similar message [ 6869.008798] Lustre: server umount lustre-OST0000 complete [ 6873.587075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.603839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6873.609402] Lustre: Skipped 1 previous similar message [ 6873.622645] Lustre: Skipped 2 previous similar messages [ 6878.692940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.698594] Lustre: Skipped 1 previous similar message [ 6879.522148] Lustre: server umount lustre-MDT0000 complete [ 6883.810830] LustreError: 197830:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6883.836528] LustreError: 197830:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6884.006684] LustreError: 197812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853852 with bad export cookie 16807430312283740026 [ 6884.009219] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6884.016384] LustreError: 197812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6884.413726] Lustre: server umount lustre-MDT0001 complete [ 6895.612064] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6898.309564] Key type lgssc unregistered [ 6898.661729] LNet: 201705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6898.676419] LNetError: 201705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6899.686796] LNet: Removed LNI 192.168.202.148@tcp [ 6900.745257] Key type .llcrypt unregistered [ 6900.749417] Key type ._llcrypt unregistered [ 6921.549405] Key type ._llcrypt registered [ 6921.551598] Key type .llcrypt registered [ 6921.691716] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 6922.718027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6922.815849] alg: No test for adler32 (adler32-zlib) [ 6923.873841] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 6924.242216] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 6925.944329] Key type lgssc registered [ 6927.073953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6935.277207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6935.308815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.909355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.394236] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6946.280726] LustreError: 202971:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.326220] LustreError: 202971:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6950.963180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.388095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6958.475792] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6966.838120] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.959498] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6980.120532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6980.614721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.617212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6987.076180] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 6994.861594] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6999.926584] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 7010.807600] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7018.069920] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 7023.371152] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 7037.921789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.922693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.929854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7037.937641] Lustre: Skipped 1 previous similar message [ 7037.959534] Lustre: Skipped 1 previous similar message [ 7042.242478] Lustre: server umount lustre-OST0000 complete [ 7048.161969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.184349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7048.189691] Lustre: Skipped 1 previous similar message [ 7053.283183] LustreError: 203891:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.308079] LustreError: 203891:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7054.456830] Lustre: server umount lustre-MDT0000 complete [ 7058.402959] LustreError: 202972:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.422182] LustreError: 202972:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7058.890383] LustreError: 206047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854027 with bad export cookie 12995101319890568888 [ 7058.902402] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7058.906260] LustreError: 206047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7059.319745] Lustre: server umount lustre-MDT0001 complete [ 7072.033156] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7075.586843] Key type lgssc unregistered [ 7075.825866] LNet: 207899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7075.835170] LNetError: 207899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7075.862923] LNet: Removed LNI 192.168.202.148@tcp [ 7076.875097] Key type .llcrypt unregistered [ 7076.877875] Key type ._llcrypt unregistered [ 7105.227962] Key type ._llcrypt registered [ 7105.229941] Key type .llcrypt registered [ 7105.352749] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7106.729252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.783204] alg: No test for adler32 (adler32-zlib) [ 7107.844311] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7108.057723] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7109.736186] Key type lgssc registered [ 7110.750954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7118.336575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7118.374523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.932733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.949667] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7129.059087] LustreError: 209317:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.079274] LustreError: 209317:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7134.184562] LustreError: 209316:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.336583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.697587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.020380] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7147.413949] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7155.493802] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7162.756051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.146138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.231552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7169.211602] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7176.825678] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.106752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.113927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.119996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7195.125025] Lustre: Skipped 1 previous similar message [ 7195.151557] Lustre: Skipped 1 previous similar message [ 7197.133080] Lustre: server umount lustre-OST0000 complete [ 7201.250150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.255306] LustreError: Skipped 1 previous similar message [ 7201.260701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.278175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.345212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.348558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.693963] Lustre: server umount lustre-MDT0000 complete [ 7210.466260] LustreError: 209316:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.474533] LustreError: 209316:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7210.760091] LustreError: 209298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854179 with bad export cookie 16875533495840596277 [ 7210.768617] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.776270] LustreError: 209298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7211.241794] Lustre: server umount lustre-MDT0001 complete [ 7223.378178] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7225.914307] Key type lgssc unregistered [ 7226.161978] LNet: 213003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7226.170520] LNetError: 213003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7226.188251] LNet: Removed LNI 192.168.202.148@tcp [ 7227.037592] Key type .llcrypt unregistered [ 7227.040311] Key type ._llcrypt unregistered [ 7246.101527] Key type ._llcrypt registered [ 7246.103601] Key type .llcrypt registered [ 7246.219872] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7247.530617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7247.561704] alg: No test for adler32 (adler32-zlib) [ 7248.604646] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7248.853641] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7250.552289] Key type lgssc registered [ 7251.907427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7259.453609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7259.480597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7260.795743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.725692] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7270.370282] LustreError: 214273:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.382228] LustreError: 214273:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7273.883103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7274.168505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.993894] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7285.363487] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7290.664405] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7296.977426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7297.497080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.361789] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7305.649681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7312.092533] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7330.273240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7330.275780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7330.282801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.300633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7331.304995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7331.326518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7333.208811] Lustre: server umount lustre-OST0000 complete [ 7341.027407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7341.038161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.059796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7341.073914] Lustre: Skipped 1 previous similar message [ 7343.464540] Lustre: server umount lustre-MDT0000 complete [ 7346.658806] LustreError: 215000:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.673571] LustreError: 215000:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7347.233234] LustreError: 214252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854316 with bad export cookie 9369059323753806864 [ 7347.238340] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7347.248213] LustreError: 214252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7347.726804] Lustre: server umount lustre-MDT0001 complete [ 7359.794471] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7362.438739] Key type lgssc unregistered [ 7362.788926] LNet: 217955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.803468] LNetError: 217955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7363.814718] LNet: Removed LNI 192.168.202.148@tcp [ 7364.740268] Key type .llcrypt unregistered [ 7364.742684] Key type ._llcrypt unregistered [ 7384.231518] Key type ._llcrypt registered [ 7384.235482] Key type .llcrypt registered [ 7384.370569] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7385.888815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7385.987705] alg: No test for adler32 (adler32-zlib) [ 7386.984426] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7387.239364] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7388.888299] Key type lgssc registered [ 7390.120424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7397.311951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7397.333462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7398.761325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.185594] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7408.099868] LustreError: 219223:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.127289] LustreError: 219223:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7412.113211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7412.452701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.258751] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7424.773411] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7431.783441] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7439.245339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.658614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.787585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7445.743971] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7453.835458] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7462.155968] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7462.216112] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7462.367309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7467.983795] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7475.636670] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7477.555633] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7489.304974] Lustre: Permanently deactivating lustre-OST0001 [ 7489.327090] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7497.796442] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7503.479435] Lustre: Permanently reactivating lustre-OST0001 [ 7503.510152] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7503.518620] Lustre: Skipped 2 previous similar messages [ 7507.512861] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7507.527188] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7507.535317] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7507.552258] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7507.595702] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7518.182522] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.187921] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.193595] Lustre: Skipped 1 previous similar message [ 7518.197254] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7518.764855] Lustre: server umount lustre-OST0001 complete [ 7523.811356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.813483] LustreError: 221044:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.835724] Lustre: Skipped 1 previous similar message [ 7523.851150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.855783] LustreError: 221044:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7523.881160] Lustre: Skipped 2 previous similar messages [ 7528.387242] Lustre: server umount lustre-OST0000 complete [ 7534.051221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.059696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7534.066669] Lustre: Skipped 2 previous similar messages [ 7538.876761] Lustre: server umount lustre-MDT0000 complete [ 7539.171870] LustreError: 219222:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.187279] LustreError: 219222:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7542.606937] LustreError: 219201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854511 with bad export cookie 7802629631285743532 [ 7542.607766] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.613856] LustreError: 219201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7542.847670] Lustre: server umount lustre-MDT0001 complete [ 7553.709572] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7556.468843] Key type lgssc unregistered [ 7556.738562] LNet: 224293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7556.743992] LNetError: 224293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7557.801357] LNet: Removed LNI 192.168.202.148@tcp [ 7558.526158] Key type .llcrypt unregistered [ 7558.528056] Key type ._llcrypt unregistered [ 7595.949607] Key type ._llcrypt registered [ 7595.952922] Key type .llcrypt registered [ 7596.047636] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7596.927435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7596.990180] alg: No test for adler32 (adler32-zlib) [ 7598.081789] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7598.441336] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7600.099269] Key type lgssc registered [ 7601.299903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7608.936365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7608.956642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7610.263859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7610.299945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7610.507784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.241176] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7625.188393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.312493] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7625.351805] Lustre: 226495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7625.643626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7629.896655] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7637.902718] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7644.574876] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7651.418338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7651.582801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7651.737941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7658.341668] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7659.733989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7665.930630] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7669.814445] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7673.164677] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7680.973850] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7687.533866] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7695.582509] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7711.695450] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7711.701837] Lustre: Skipped 1 previous similar message [ 7747.618434] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7747.629302] Lustre: Skipped 3 previous similar messages [ 7812.492370] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7812.497765] Lustre: Skipped 7 previous similar messages [ 7854.566953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7854.567191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7854.579981] LustreError: Skipped 1 previous similar message [ 7854.584692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7854.588991] Lustre: Skipped 1 previous similar message [ 7857.849349] Lustre: server umount lustre-OST0000 complete [ 7864.801437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.817104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7864.822161] Lustre: Skipped 2 previous similar messages [ 7869.922209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7869.930458] Lustre: Skipped 1 previous similar message [ 7875.042347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7875.047379] Lustre: Skipped 1 previous similar message [ 7876.064226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7876.307914] Lustre: server umount lustre-MDT0000 complete [ 7880.240279] LustreError: 226488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854849 with bad export cookie 309176136160253067 [ 7880.247889] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.257143] LustreError: 226488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7880.669076] Lustre: server umount lustre-MDT0001 complete [ 7891.342931] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 7893.880050] Key type lgssc unregistered [ 7894.219507] LNet: 231320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7894.226757] LNetError: 231320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.247936] LNet: Removed LNI 192.168.202.148@tcp [ 7895.050811] Key type .llcrypt unregistered [ 7895.052185] Key type ._llcrypt unregistered [ 7917.759682] Key type ._llcrypt registered [ 7917.761471] Key type .llcrypt registered [ 7917.851321] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 7920.359844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7920.387769] alg: No test for adler32 (adler32-zlib) [ 7921.467801] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 7921.775160] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 7923.505392] Key type lgssc registered [ 7924.820920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7932.739531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7932.773359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.274812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.996670] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7944.674605] LustreError: 232732:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7944.693802] LustreError: 232732:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7948.454579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.773093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7953.646703] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7962.075137] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7968.799967] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7975.539720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.828628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.997724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7981.738515] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug -1 all [ 7990.863993] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8008.681959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8008.694705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.699031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.751749] Lustre: Skipped 1 previous similar message [ 8013.803260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8013.810847] Lustre: Skipped 1 previous similar message [ 8014.750765] Lustre: server umount lustre-OST0000 complete [ 8018.918530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.922293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8018.943401] Lustre: Skipped 1 previous similar message [ 8018.965497] Lustre: Skipped 2 previous similar messages [ 8024.033861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8024.043981] Lustre: Skipped 1 previous similar message [ 8024.930823] Lustre: server umount lustre-MDT0000 complete [ 8028.758225] LustreError: 232712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854997 with bad export cookie 6236160262777922612 [ 8028.762244] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8028.775593] LustreError: 232712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8029.204383] Lustre: server umount lustre-MDT0001 complete [ 8040.706099] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8044.226821] Key type lgssc unregistered [ 8044.555203] LNet: 236428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.567331] LNetError: 236428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.591783] LNet: Removed LNI 192.168.202.148@tcp [ 8045.495511] Key type .llcrypt unregistered [ 8045.500606] Key type ._llcrypt unregistered [ 8074.906669] Key type ._llcrypt registered [ 8074.913600] Key type .llcrypt registered [ 8075.216503] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8077.014074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8077.121104] alg: No test for adler32 (adler32-zlib) [ 8078.742408] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8079.260103] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8081.097729] Key type lgssc registered [ 8082.687477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8353.920552] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8356.374952] Key type lgssc unregistered [ 8356.586597] LNet: 238453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8356.592537] LNetError: 238453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8356.603277] LNet: Removed LNI 192.168.202.148@tcp [ 8357.373233] Key type .llcrypt unregistered [ 8357.377414] Key type ._llcrypt unregistered [ 8384.448777] Key type ._llcrypt registered [ 8384.450670] Key type .llcrypt registered [ 8384.579557] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 8403.114876] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 8403.986774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8404.029164] alg: No test for adler32 (adler32-zlib) [ 8405.122904] Lustre: Lustre: Build Version: 2.17.57_87_ge30c52f [ 8405.431934] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 8407.064135] Key type lgssc registered [ 8408.097817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8440.613926] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8106 sec ========== 14:30:08 (1787855408) [ 8442.203792] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:30:10 (1787855410) === [ 8446.404433] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:30:13 (1787855413) === [ 8486.639659] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 8490.015822] Key type lgssc unregistered [ 8490.274770] LNet: 243710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8490.284437] LNetError: 243710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8490.302735] LNet: Removed LNI 192.168.202.148@tcp [ 8491.225621] Key type .llcrypt unregistered [ 8491.229596] Key type ._llcrypt unregistered