[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508608845 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007017] pid_max: default: 32768 minimum: 301 [ 0.008174] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012099] *** VALIDATE selinux *** [ 0.022019] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027835] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031439] *** VALIDATE proc *** [ 0.033179] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035145] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037026] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041763] debug: unmapping init [mem 0xffffffffa0459000-0xffffffffa0460fff] [ 0.043885] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044614] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.045994] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051259] rcu: Hierarchical SRCU implementation. [ 0.053423] smp: Bringing up secondary CPUs ... [ 0.054601] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.058355] smp: Brought up 1 node, 4 CPUs [ 0.059974] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133092] node 0 deferred pages initialised in 72ms [ 0.136112] devtmpfs: initialized [ 0.137213] x86/mm: Memory block size: 128MB [ 0.140000] gcov: version magic: 0x41383552 [ 0.143404] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150335] pinctrl core: initialized pinctrl subsystem [ 0.152184] [ 0.154012] ************************************************************* [ 0.156015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159016] ** ** [ 0.162019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166015] ** ** [ 0.170016] ** This means that this kernel is built to expose internal ** [ 0.173014] ** IOMMU data structures, which may compromise security on ** [ 0.177017] ** your system. ** [ 0.180011] ** ** [ 0.184015] ** If you see this message and you are not debugging the ** [ 0.187013] ** kernel, report this immediately to your vendor! ** [ 0.191015] ** ** [ 0.194018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198016] ************************************************************* [ 0.201785] NET: Registered protocol family 16 [ 0.203452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.209075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213013] cpuidle: using governor menu [ 0.214724] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217495] PCI: Using configuration type 1 for base access [ 0.218135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.229130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233056] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237133] cryptd: max_cpu_qlen set to 1000 [ 0.239024] ACPI: Added _OSI(Module Device) [ 0.241016] ACPI: Added _OSI(Processor Device) [ 0.244016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252025] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260599] ACPI: Interpreter enabled [ 0.262046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264015] ACPI: Using IOAPIC for interrupt routing [ 0.266161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281916] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.287039] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.290113] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.295664] acpiphp: Slot [2] registered [ 0.297142] acpiphp: Slot [5] registered [ 0.298148] acpiphp: Slot [6] registered [ 0.300169] acpiphp: Slot [7] registered [ 0.301113] acpiphp: Slot [8] registered [ 0.303123] acpiphp: Slot [9] registered [ 0.304105] acpiphp: Slot [10] registered [ 0.306069] acpiphp: Slot [3] registered [ 0.307098] acpiphp: Slot [4] registered [ 0.309097] acpiphp: Slot [11] registered [ 0.311117] acpiphp: Slot [12] registered [ 0.312082] acpiphp: Slot [13] registered [ 0.313094] acpiphp: Slot [14] registered [ 0.314117] acpiphp: Slot [15] registered [ 0.316110] acpiphp: Slot [16] registered [ 0.318112] acpiphp: Slot [17] registered [ 0.320104] acpiphp: Slot [18] registered [ 0.321114] acpiphp: Slot [19] registered [ 0.322101] acpiphp: Slot [20] registered [ 0.324128] acpiphp: Slot [21] registered [ 0.325086] acpiphp: Slot [22] registered [ 0.327124] acpiphp: Slot [23] registered [ 0.329175] acpiphp: Slot [24] registered [ 0.330111] acpiphp: Slot [25] registered [ 0.332123] acpiphp: Slot [26] registered [ 0.333215] acpiphp: Slot [27] registered [ 0.335110] acpiphp: Slot [28] registered [ 0.337092] acpiphp: Slot [29] registered [ 0.338150] acpiphp: Slot [30] registered [ 0.340119] acpiphp: Slot [31] registered [ 0.342069] PCI host bridge to bus 0000:00 [ 0.343022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360940] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373854] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.382039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.392127] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395040] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.401678] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.417017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.420013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424649] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.434016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.465015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474911] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517587] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.524016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.531017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.557716] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.565017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.597018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.608220] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.615020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.660178] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.675013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.695016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.708142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710318] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720031] iommu: Default domain type: Passthrough [ 0.721408] SCSI subsystem initialized [ 0.722127] ACPI: bus type USB registered [ 0.723104] usbcore: registered new interface driver usbfs [ 0.725069] usbcore: registered new interface driver hub [ 0.726070] usbcore: registered new device driver usb [ 0.727154] pps_core: LinuxPPS API ver. 1 registered [ 0.729008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.731055] PTP clock support registered [ 0.734042] EDAC MC: Ver: 3.0.0 [ 0.736150] PCI: Using ACPI for IRQ routing [ 0.737860] NetLabel: Initializing [ 0.739011] NetLabel: domain hash size = 128 [ 0.740010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.741110] NetLabel: unlabeled traffic allowed by default [ 0.743042] vgaarb: loaded [ 0.744281] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750410] clocksource: Switched to clocksource kvm-clock [ 0.857659] VFS: Disk quotas dquot_6.6.0 [ 0.859355] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.861988] *** VALIDATE ramfs *** [ 0.863519] *** VALIDATE hugetlbfs *** [ 0.864755] pnp: PnP ACPI init [ 0.867075] pnp: PnP ACPI: found 6 devices [ 0.885295] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.887978] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.889529] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.891230] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.893454] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.895613] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.898481] NET: Registered protocol family 2 [ 0.900856] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.905563] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.908875] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.913967] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.917253] TCP: Hash tables configured (established 65536 bind 65536) [ 0.919912] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.922722] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925144] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927870] NET: Registered protocol family 1 [ 0.930042] RPC: Registered named UNIX socket transport module. [ 0.932524] RPC: Registered udp transport module. [ 0.934369] RPC: Registered tcp transport module. [ 0.936317] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.938916] NET: Registered protocol family 44 [ 0.943672] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.946080] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.948302] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.950715] PCI: CLS 0 bytes, default 64 [ 0.952979] Unpacking initramfs... [ 2.385941] debug: unmapping init [mem 0xffff99e9fcc54000-0xffff99e9fffbffff] [ 2.390403] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.392690] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.396219] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.912295] Initialise system trusted keyrings [ 2.913989] Key type blacklist registered [ 2.916442] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.927057] zbud: loaded [ 2.930555] *** VALIDATE nfs *** [ 2.931983] *** VALIDATE nfs4 *** [ 2.933683] pstore: using deflate compression [ 2.937825] Platform Keyring initialized [ 3.044347] NET: Registered protocol family 38 [ 3.046440] Key type asymmetric registered [ 3.047988] Asymmetric key parser 'x509' registered [ 3.049990] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.053192] io scheduler mq-deadline registered [ 3.055031] io scheduler kyber registered [ 3.056674] io scheduler bfq registered [ 3.059038] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.062253] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.064549] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.067828] ACPI: Power Button [PWRF] [ 3.073494] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.079971] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.092855] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.099480] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.119394] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.147551] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.176695] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.181857] Non-volatile memory driver v1.3 [ 3.183699] Linux agpgart interface v0.103 [ 3.215364] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.218524] vda: detected capacity change from 0 to 74698752 [ 3.232664] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.236090] vdb: detected capacity change from 0 to 1073741824 [ 3.250875] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.253973] vdc: detected capacity change from 0 to 2621440000 [ 3.267883] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.270551] vdd: detected capacity change from 0 to 2621440000 [ 3.285596] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.288957] vde: detected capacity change from 0 to 4294967296 [ 3.305823] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308965] vdf: detected capacity change from 0 to 4294967296 [ 3.316549] libphy: Fixed MDIO Bus: probed [ 3.325181] usbcore: registered new interface driver usbserial_generic [ 3.327845] usbserial: USB Serial support registered for generic [ 3.330285] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.334787] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.336698] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.339399] mousedev: PS/2 mouse device common for all mice [ 3.342606] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.344540] rtc_cmos 00:05: RTC can wake from S4 [ 3.349869] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.350176] rtc_cmos 00:05: registered as rtc0 [ 3.355495] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.358993] intel_pstate: CPU model not supported [ 3.362718] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.367387] hid: raw HID events driver (C) Jiri Kosina [ 3.369832] usbcore: registered new interface driver usbhid [ 3.372171] usbhid: USB HID core driver [ 3.373931] drop_monitor: Initializing network drop monitor service [ 3.376669] Initializing XFRM netlink socket [ 3.378592] NET: Registered protocol family 10 [ 3.382650] Segment Routing with IPv6 [ 3.384450] NET: Registered protocol family 17 [ 3.387026] mpls_gso: MPLS GSO support [ 3.392839] RAS: Correctable Errors collector initialized. [ 3.395034] AVX version of gcm_enc/dec engaged. [ 3.397283] AES CTR mode by8 optimization enabled [ 3.477941] sched_clock: Marking stable (3477923158, 0)->(4378680324, -900757166) [ 3.482158] registered taskstats version 1 [ 3.484254] Loading compiled-in X.509 certificates [ 3.486254] zswap: loaded using pool lzo/zbud [ 3.511879] Key type big_key registered [ 3.524284] Key type encrypted registered [ 3.525521] ima: No TPM chip found, activating TPM-bypass! [ 3.527784] ima: Allocated hash algorithm: sha1 [ 3.530139] ima: No architecture policies found [ 3.531488] evm: Initialising EVM extended attributes: [ 3.533049] evm: security.selinux [ 3.534338] evm: security.ima [ 3.535290] evm: security.capability [ 3.536292] evm: HMAC attrs: 0x1 [ 3.538352] rtc_cmos 00:05: setting system clock to 2026-09-03 13:13:32 UTC (1788441212) [ 3.544167] debug: unmapping init [mem 0xffffffffa1403000-0xffffffffa15fffff] [ 3.547350] debug: unmapping init [mem 0xffffffffa0182000-0xffffffffa0458fff] [ 3.556225] Write protecting the kernel read-only data: 28672k [ 3.559389] debug: unmapping init [mem 0xffffffff9e803000-0xffffffff9e9fffff] [ 3.561923] debug: unmapping init [mem 0xffffffff9f114000-0xffffffff9f1fffff] [ 3.595520] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.605110] systemd[1]: Detected virtualization kvm. [ 3.607096] systemd[1]: Detected architecture x86-64. [ 3.609255] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.635836] systemd[1]: No hostname configured. [ 3.637592] systemd[1]: Set hostname to . [ 3.639312] random: systemd: uninitialized urandom read (16 bytes read) [ 3.641977] systemd[1]: Initializing machine ID from random generator. [ 3.689056] random: ln: uninitialized urandom read (6 bytes read) [ 3.769833] random: systemd: uninitialized urandom read (16 bytes read) [ 3.773135] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.778128] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.782510] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.369675] device-mapper: uevent: version 1.0.3 [ 4.371977] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.070364] virtio_net virtio0 ens2: renamed from eth0 [ 5.097461] scsi host0: ata_piix [ 5.123910] random: fast init done [ 5.168791] scsi host1: ata_piix [ 5.170604] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.172907] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.720964] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.059529] random: crng init done [ 10.061734] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.391647] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.485426] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.744187] SELinux: Disabled at runtime. [ 11.809068] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.817591] systemd[1]: Detected virtualization kvm. [ 11.819319] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.345517] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.349355] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.354286] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.359262] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.363876] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.371214] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.374592] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [[ 12.528594] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.813517] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.107847] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.221216] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.319892] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.331609] EDAC sbridge: Ver: 1.1.2 [ 14.543546] Key type dns_resolver registered [ 14.869456] NFS: Registering the id_resolver key type [ 14.873991] Key type id_resolver registered [ 14.876228] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-server login: [ 44.249652] hrtimer: interrupt took 6017156 ns [ 44.701519] libcfs: loading out-of-tree module taints kernel. [ 44.860427] Key type ._llcrypt registered [ 44.864195] Key type .llcrypt registered [ 45.105611] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 67.131075] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 68.930075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.963963] alg: No test for adler32 (adler32-zlib) [ 70.595481] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 72.080353] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 74.143526] Key type lgssc registered [ 76.266396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.446179] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.435820] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 161.393329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.491173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.924516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.976053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.085216] Lustre: lustre-MDT0000: new disk, initializing [ 163.238343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.286204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.129052] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.654347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.825249] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.875231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.884261] Lustre: Skipped 1 previous similar message [ 183.029213] Lustre: lustre-MDT0001: new disk, initializing [ 183.142190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.177095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.184201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.937736] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.921780] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.619868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.890456] Lustre: lustre-OST0000: new disk, initializing [ 203.894897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.901469] Lustre: 8471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.968837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.454798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.470401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.613357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.525158] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.031566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.222211] Lustre: lustre-OST0001: new disk, initializing [ 228.229660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.240373] Lustre: 9544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.340709] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.644894] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.084644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.096226] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.158681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 249.316860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.771808] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.368257] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.275728] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing yml_node [ 272.347612] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 274.955605] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 277.510910] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 279.208512] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:18:06 EDT 2026 [ 298.087533] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 299.755424] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.386724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.393336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.416507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.944196] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.945116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.959935] Lustre: Skipped 2 previous similar messages [ 318.989748] Lustre: Skipped 3 previous similar messages [ 324.318397] Lustre: server umount lustre-MDT0000 complete [ 328.535329] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441537 with bad export cookie 14627165154090425606 [ 328.539652] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.546515] LustreError: 6525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 329.186927] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.188730] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.206638] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.228971] LustreError: 6539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 329.251731] Lustre: Skipped 1 previous similar message [ 333.217248] LustreError: 6540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.219376] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.250404] Lustre: Skipped 1 previous similar message [ 334.894249] Lustre: server umount lustre-MDT0001 complete [ 338.563636] Lustre: server umount lustre-OST0000 complete [ 342.772678] Lustre: server umount lustre-OST0001 complete [ 350.676123] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 362.431350] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 402.833291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.118114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.151262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.247440] Lustre: lustre-MDT0000: new disk, initializing [ 403.392638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.418084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.984365] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 418.535468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.655514] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 418.715846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 418.722839] Lustre: Skipped 1 previous similar message [ 418.815901] Lustre: lustre-MDT0001: new disk, initializing [ 418.907271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 418.948846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 418.959379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 423.410425] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 433.509754] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 439.211535] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 446.533594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 446.985888] Lustre: lustre-OST0000: new disk, initializing [ 446.999077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.019749] Lustre: 17063:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.130324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.616272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 448.637965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 448.816247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 453.612143] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 463.783554] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.636248] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.973502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.358097] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.670616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 479.712702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.720897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.730727] LustreError: Skipped 1 previous similar message [ 479.735626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.745789] Lustre: Skipped 2 previous similar messages [ 489.957907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.965921] Lustre: Skipped 4 previous similar messages [ 492.001337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 492.138625] Lustre: server umount lustre-OST0000 complete [ 500.197627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.218413] Lustre: Skipped 1 previous similar message [ 502.289230] Lustre: server umount lustre-MDT0000 complete [ 505.313043] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.366893] LustreError: 15106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 507.030829] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441715 with bad export cookie 14627165154090427139 [ 507.034609] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.053148] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 507.608963] Lustre: server umount lustre-MDT0001 complete [ 512.037813] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:21:59 (1788441719) [ 519.347286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.926552] LustreError: 19329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.011319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.026767] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 525.282629] LustreError: 19330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.504164] LustreError: 19329:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.533482] LustreError: 19329:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 534.658501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.237363] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 546.259751] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.460670] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 561.943163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.693823] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 577.145856] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 588.778811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 588.789394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.817351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.824374] Lustre: Skipped 2 previous similar messages [ 593.174490] Lustre: server umount lustre-OST0000 complete [ 602.805517] Lustre: server umount lustre-MDT0000 complete [ 605.669517] LustreError: 21819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.710843] LustreError: 21819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 606.843347] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441815 with bad export cookie 14627165154090428021 [ 606.847632] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.866196] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 607.333178] Lustre: server umount lustre-MDT0001 complete [ 618.913617] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 622.232952] Key type lgssc unregistered [ 622.617298] LNet: 22862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.624614] LNetError: 22862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.653991] LNet: Removed LNI 192.168.204.135@tcp [ 624.627175] Key type .llcrypt unregistered [ 624.631293] Key type ._llcrypt unregistered [ 647.146559] Key type ._llcrypt registered [ 647.148706] Key type .llcrypt registered [ 647.298826] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 648.554471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.565901] alg: No test for adler32 (adler32-zlib) [ 649.691369] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 649.974392] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 651.712613] Key type lgssc registered [ 652.922702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 661.115391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.137507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.595631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.596734] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 672.737273] LustreError: 24274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.764476] LustreError: 24274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 676.509653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.866716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 681.523174] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 689.267919] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 695.318757] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 702.243769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 702.774789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.924426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 709.950723] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 717.898794] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 736.089864] Lustre: server umount lustre-OST0000 complete [ 741.861166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.873756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.876924] Lustre: Skipped 1 previous similar message [ 741.885171] Lustre: Skipped 1 previous similar message [ 746.978725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 746.988915] Lustre: Skipped 1 previous similar message [ 750.106829] Lustre: server umount lustre-MDT0000 complete [ 752.096504] LustreError: 24274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 752.099396] Lustre: 23586:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441945/real 1788441945] req@00000000f1efc414 x1875317209722880/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441961 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 752.122038] LustreError: 24274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 752.160793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.832811] LustreError: 24254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441962 with bad export cookie 106557656417814731 [ 753.838986] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 753.845865] LustreError: 24254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 754.126827] Lustre: server umount lustre-MDT0001 complete [ 766.655332] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 769.715131] Key type lgssc unregistered [ 770.014051] LNet: 28254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.035336] LNetError: 28254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.049366] LNet: Removed LNI 192.168.204.135@tcp [ 770.890856] Key type .llcrypt unregistered [ 770.902967] Key type ._llcrypt unregistered [ 795.326538] Key type ._llcrypt registered [ 795.329597] Key type .llcrypt registered [ 795.463356] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 796.430625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.478524] alg: No test for adler32 (adler32-zlib) [ 797.608848] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 797.852679] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 799.551510] Key type lgssc registered [ 800.923501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 809.761944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 809.823870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 811.529874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.351063] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 821.731438] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.756319] LustreError: 29670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 824.894342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.247465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 828.933594] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 836.233225] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 842.790145] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 858.074944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 858.351865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.741349] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 871.269524] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 881.913660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 891.360658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 891.369233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 891.389324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 892.390072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.396483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 894.656101] Lustre: server umount lustre-OST0000 complete [ 902.112432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 902.131748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 902.147952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 902.155336] Lustre: Skipped 1 previous similar message [ 907.750059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.756797] Lustre: Skipped 2 previous similar messages [ 912.864316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 913.376759] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 913.829168] Lustre: server umount lustre-MDT0000 complete [ 917.886313] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442126 with bad export cookie 558386610675784438 [ 917.892431] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.900658] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 918.602239] Lustre: server umount lustre-MDT0001 complete [ 930.666605] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 933.865547] Key type lgssc unregistered [ 934.194601] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.204232] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.224867] LNet: Removed LNI 192.168.204.135@tcp [ 935.163100] Key type .llcrypt unregistered [ 935.167934] Key type ._llcrypt unregistered [ 960.095607] Key type ._llcrypt registered [ 960.104798] Key type .llcrypt registered [ 960.306844] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 961.590109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 961.763879] alg: No test for adler32 (adler32-zlib) [ 962.800341] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 963.028603] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 964.751736] Key type lgssc registered [ 965.974638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 976.086474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 976.112507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.687635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.702615] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 987.109663] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.125449] LustreError: 34991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 990.484190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.963169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.487224] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1004.259266] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1010.942787] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.307314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.751382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1022.837430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1026.338583] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1035.405454] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1045.472906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1045.481653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.487526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1045.499801] Lustre: Skipped 1 previous similar message [ 1045.515707] Lustre: Skipped 1 previous similar message [ 1050.417878] Lustre: server umount lustre-OST0000 complete [ 1055.723515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.725147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.734645] Lustre: Skipped 1 previous similar message [ 1055.750423] Lustre: Skipped 1 previous similar message [ 1060.597693] Lustre: server umount lustre-MDT0000 complete [ 1060.833141] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1060.875030] LustreError: 34986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1064.501528] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442273 with bad export cookie 17202044670816945398 [ 1064.518509] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.523286] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1064.900710] Lustre: server umount lustre-MDT0001 complete [ 1077.191633] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1081.361138] Key type lgssc unregistered [ 1081.707524] LNet: 38529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.712992] LNetError: 38529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.728056] LNet: Removed LNI 192.168.204.135@tcp [ 1082.739114] Key type .llcrypt unregistered [ 1082.743423] Key type ._llcrypt unregistered [ 1109.234254] Key type ._llcrypt registered [ 1109.236930] Key type .llcrypt registered [ 1109.344480] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1110.766752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1110.806378] alg: No test for adler32 (adler32-zlib) [ 1111.919973] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1112.289979] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1114.039195] Key type lgssc registered [ 1115.355666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1123.605704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1123.649886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.331685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.796435] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1135.585428] LustreError: 39943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.604785] LustreError: 39943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1138.111748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.564043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1143.043963] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1151.926298] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1158.275533] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1166.351404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1166.733116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.820408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1172.834959] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1180.456971] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1191.400865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.417538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.467634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.456806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.459583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.481860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.569847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.578625] Lustre: Skipped 2 previous similar messages [ 1203.687100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.703782] Lustre: Skipped 2 previous similar messages [ 1205.215186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1205.432322] Lustre: server umount lustre-OST0000 complete [ 1229.283838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.287462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1229.320962] Lustre: Skipped 1 previous similar message [ 1229.325530] Lustre: Skipped 1 previous similar message [ 1232.649552] Lustre: server umount lustre-MDT0000 complete [ 1233.442536] LustreError: 40670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.468100] LustreError: 40670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1236.702690] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442445 with bad export cookie 6571027692673122655 [ 1236.707809] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.715538] LustreError: 39923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1251.295408] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1251.708128] Lustre: server umount lustre-MDT0001 complete [ 1263.986657] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1267.150315] Key type lgssc unregistered [ 1267.507159] LNet: 43655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1267.526271] LNetError: 43655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.543289] LNet: Removed LNI 192.168.204.135@tcp [ 1268.719220] Key type .llcrypt unregistered [ 1268.725417] Key type ._llcrypt unregistered [ 1294.351119] Key type ._llcrypt registered [ 1294.354085] Key type .llcrypt registered [ 1294.472739] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1295.770265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.842129] alg: No test for adler32 (adler32-zlib) [ 1297.002959] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1297.293836] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1298.967299] Key type lgssc registered [ 1300.215549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1307.690146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1307.722651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.141730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.953736] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1319.399281] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.411567] LustreError: 45069:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1322.739435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.168519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1327.707864] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1336.320618] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.879204] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.838379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.194374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.313446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1357.407386] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1365.152649] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1376.919341] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1378.793479] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.797675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.809181] Lustre: Skipped 2 previous similar messages [ 1378.827806] Lustre: Skipped 2 previous similar messages [ 1380.596034] Lustre: server umount lustre-MDT0000 complete [ 1382.019378] LustreError: 47105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.906544] LustreError: 45070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.920405] LustreError: 45070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1384.775296] LustreError: 45050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442593 with bad export cookie 566362381483194129 [ 1384.784038] LustreError: 45050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1384.784427] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.131664] LustreError: 45065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.186946] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [ 1389.030020] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.252262] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.256285] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [ 1392.279466] LustreError: 45064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1392.304836] Lustre: Skipped 1 previous similar message [ 1397.375385] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [ 1397.384087] Lustre: Skipped 1 previous similar message [ 1399.265962] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1399.527275] Lustre: server umount lustre-MDT0001 complete [ 1414.664074] Lustre: server umount lustre-OST0000 complete [ 1431.059939] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1433.837957] Key type lgssc unregistered [ 1434.167815] LNet: 48930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.179853] LNetError: 48930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.207770] LNet: Removed LNI 192.168.204.135@tcp [ 1435.235554] Key type .llcrypt unregistered [ 1435.238074] Key type ._llcrypt unregistered [ 1461.155828] Key type ._llcrypt registered [ 1461.158811] Key type .llcrypt registered [ 1461.259958] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1462.675565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.743668] alg: No test for adler32 (adler32-zlib) [ 1463.788283] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1463.967720] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1465.655297] Key type lgssc registered [ 1466.830655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1474.007557] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1474.035372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.631448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1494.751894] LustreError: 50318:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1521.439236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.880406] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1535.408349] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1699.598758] Lustre: server umount lustre-OST0000 complete [ 1716.179746] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1720.059664] Key type lgssc unregistered [ 1720.502848] LNet: 51997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.524249] LNetError: 51997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.551367] LNet: Removed LNI 192.168.204.135@tcp [ 1721.656571] Key type .llcrypt unregistered [ 1721.659266] Key type ._llcrypt unregistered [ 1749.833531] Key type ._llcrypt registered [ 1749.840486] Key type .llcrypt registered [ 1749.985852] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1751.020868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.043174] alg: No test for adler32 (adler32-zlib) [ 1752.331917] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1752.636082] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1754.399403] Key type lgssc registered [ 1755.702956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.858548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1763.875311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.624224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.007819] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1775.081954] LustreError: 53411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.106516] LustreError: 53411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1780.195836] LustreError: 53412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1780.555959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.051484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1785.826932] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1794.420748] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1801.346208] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.496067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1809.026249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1811.084769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1816.189305] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1823.089217] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1826.343065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1826.351793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1831.393142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1831.406417] Lustre: Skipped 1 previous similar message [ 1831.629787] Lustre: server umount lustre-OST0000 complete [ 1836.513729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1836.518632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.538860] Lustre: Skipped 1 previous similar message [ 1836.549826] Lustre: Skipped 1 previous similar message [ 1841.633168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1841.649708] Lustre: Skipped 2 previous similar messages [ 1841.979632] Lustre: server umount lustre-MDT0000 complete [ 1845.856454] LustreError: 53393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443054 with bad export cookie 6175663308041154520 [ 1845.863187] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.868687] LustreError: 53393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1846.311336] Lustre: server umount lustre-MDT0001 complete [ 1858.486486] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1861.982455] Key type lgssc unregistered [ 1862.413084] LNet: 56948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.447498] LNetError: 56948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1862.477635] LNet: Removed LNI 192.168.204.135@tcp [ 1863.894358] Key type .llcrypt unregistered [ 1863.896737] Key type ._llcrypt unregistered [ 1890.629261] Key type ._llcrypt registered [ 1890.633513] Key type .llcrypt registered [ 1890.796113] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1891.793832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.879027] alg: No test for adler32 (adler32-zlib) [ 1893.227534] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1893.503470] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1895.175278] Key type lgssc registered [ 1896.634321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1906.125828] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1906.162483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.828918] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1926.947160] LustreError: 58335:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1953.695213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1962.201238] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1971.308768] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.406406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.236114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.379928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1985.411564] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1985.522416] LustreError: 58361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.554650] LustreError: 58361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1988.583552] LustreError: 59371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.632014] LustreError: 59371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1993.704887] LustreError: 58361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.742428] LustreError: 58361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1996.068729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.810840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.085678] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2010.604126] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2017.269023] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2021.344672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.360330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.391038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2022.370729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.375812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.377512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.801288] Lustre: server umount lustre-OST0000 complete [ 2053.089938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.105521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2053.124603] Lustre: Skipped 1 previous similar message [ 2053.127358] Lustre: Skipped 1 previous similar message [ 2053.139125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.997682] Lustre: server umount lustre-MDT0000 complete [ 2058.209350] LustreError: 59371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.269969] LustreError: 59371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2060.928406] LustreError: 58343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443269 with bad export cookie 15931869095449234258 [ 2060.936414] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2060.940077] LustreError: 58343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2067.255475] Lustre: server umount lustre-MDT0001 complete [ 2078.541369] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2081.980038] Key type lgssc unregistered [ 2082.291901] LNet: 62082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.306526] LNetError: 62082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.334747] LNet: Removed LNI 192.168.204.135@tcp [ 2083.375553] Key type .llcrypt unregistered [ 2083.379114] Key type ._llcrypt unregistered [ 2106.942445] Key type ._llcrypt registered [ 2106.947882] Key type .llcrypt registered [ 2107.166662] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2108.521914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.633600] alg: No test for adler32 (adler32-zlib) [ 2109.750128] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2110.058091] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2111.975194] Key type lgssc registered [ 2113.854409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2123.133416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2123.158846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.746303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.891227] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2135.008869] LustreError: 63494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.019165] LustreError: 63494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2138.005225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.250549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.111351] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2150.035718] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.815309] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.849221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.133772] LustreError: 65317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.415212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.850292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2170.455493] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2178.474237] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2198.063071] Lustre: server umount lustre-OST0000 complete [ 2204.641415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2204.654115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2204.675582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.665912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.669558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.687674] Lustre: Skipped 1 previous similar message [ 2210.788558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.804938] Lustre: Skipped 1 previous similar message [ 2212.910573] Lustre: server umount lustre-MDT0000 complete [ 2215.839456] Lustre: 62804:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443408/real 1788443408] req@000000009bffaa8d x1875318741693824/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443424 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.870788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.898368] LustreError: 64223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.919196] LustreError: 64223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2216.927101] Lustre: 62807:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443409/real 1788443409] req@000000002d686b4b x1875318741694592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788443425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2217.531919] LustreError: 63474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443426 with bad export cookie 2967657812698609671 [ 2217.544270] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.558178] LustreError: 63474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2218.049391] Lustre: server umount lustre-MDT0001 complete [ 2231.126888] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2234.758788] Key type lgssc unregistered [ 2235.194435] LNet: 67054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.219374] LNetError: 67054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.249359] LNet: Removed LNI 192.168.204.135@tcp [ 2236.564080] Key type .llcrypt unregistered [ 2236.566286] Key type ._llcrypt unregistered [ 2266.058841] Key type ._llcrypt registered [ 2266.061507] Key type .llcrypt registered [ 2266.174921] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2267.917273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.965146] alg: No test for adler32 (adler32-zlib) [ 2269.060887] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2269.395193] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2271.087778] Key type lgssc registered [ 2272.492726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2282.333838] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:51:29 (1788443489) [ 2289.680794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2289.719863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.225030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.758392] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2301.411655] LustreError: 68761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.443093] LustreError: 68761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2306.493036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.534191] LustreError: 68760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2306.991204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.929774] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2319.651248] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2325.463817] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2333.075167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.400684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.529301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2339.441465] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2347.446968] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2357.393110] Lustre: Failing over lustre-MDT0000 [ 2357.755041] Lustre: server umount lustre-MDT0000 complete [ 2358.243882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2358.252399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.270434] LustreError: 68761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.473015] LustreError: 68756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.338920] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.570930] LustreError: 68756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.593756] LustreError: 68756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2367.396582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.530670] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.714937] Lustre: *** cfs_fail_loc=135, val=0*** [ 2367.856271] LustreError: 71847:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2367.866903] LustreError: 71847:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 2367.887800] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2367.887800] [ 2367.899062] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2367.918592] LustreError: 71835:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2367.930276] LustreError: 71835:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2367.935463] LustreError: 71835:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2367.942265] LustreError: 71835:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2367.968506] Lustre: server umount lustre-MDT0000 complete [ 2367.973827] LustreError: 71835:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2374.578605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.711286] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.988252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.042228] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2375.813200] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2379.551953] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2380.277789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2380.310942] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2380.370407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2387.939213] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.946205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.959575] Lustre: Skipped 1 previous similar message [ 2387.971726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.511496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.528887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.843737] Lustre: server umount lustre-OST0000 complete [ 2400.736958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2400.739406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.746787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.759880] Lustre: Skipped 1 previous similar message [ 2400.773827] Lustre: Skipped 2 previous similar messages [ 2403.990644] Lustre: server umount lustre-MDT0000 complete [ 2405.858892] LustreError: 69750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.874155] LustreError: 69750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2407.445918] LustreError: 68742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443616 with bad export cookie 3084687754122442629 [ 2407.452843] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.454778] LustreError: 68742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2407.961400] Lustre: server umount lustre-MDT0001 complete [ 2419.877830] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2423.143882] Key type lgssc unregistered [ 2423.493512] LNet: 73631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2423.506923] LNetError: 73631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2423.541736] LNet: Removed LNI 192.168.204.135@tcp [ 2424.431401] Key type .llcrypt unregistered [ 2424.433277] Key type ._llcrypt unregistered [ 2445.204607] Key type ._llcrypt registered [ 2445.212590] Key type .llcrypt registered [ 2445.309214] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2446.314476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.347250] alg: No test for adler32 (adler32-zlib) [ 2447.464807] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2447.719953] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2449.375508] Key type lgssc registered [ 2450.454460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.908312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.935819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.437439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.643889] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2470.882306] LustreError: 75022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.919517] LustreError: 75022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2474.152802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.447585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2478.956930] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2487.758543] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2494.260352] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2502.019178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.342738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.521054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2508.099375] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2516.011131] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2525.859124] Lustre: Failing over lustre-MDT0000 [ 2526.665036] Lustre: server umount lustre-MDT0000 complete [ 2528.939061] LustreError: 75019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.274919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2530.289377] LustreError: 75023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.302949] Lustre: Skipped 1 previous similar message [ 2530.354209] LustreError: 75023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2534.029511] LustreError: 77323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.849520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.026733] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2536.153882] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2536.293215] LustreError: 78112:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2536.303825] LustreError: 78112:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -9 [ 2536.310886] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2536.310886] [ 2536.323127] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2536.330893] LustreError: 78101:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2536.335123] LustreError: 78101:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2536.341022] LustreError: 78101:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2536.345839] LustreError: 78101:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2536.359702] Lustre: server umount lustre-MDT0000 complete [ 2536.364686] LustreError: 78101:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2539.135067] LustreError: 76028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2539.169744] LustreError: 76028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2543.244922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.419933] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2543.432327] LustreError: 75014:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2543.746494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.861235] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2544.264254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2547.529218] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2549.225535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2549.246389] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2549.317439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2555.873604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.890058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.912804] Lustre: Skipped 1 previous similar message [ 2555.934995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.461038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.461151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.723480] Lustre: server umount lustre-OST0000 complete [ 2569.697327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2569.697585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.698367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.698372] Lustre: Skipped 1 previous similar message [ 2569.724659] Lustre: Skipped 1 previous similar message [ 2572.171215] Lustre: server umount lustre-MDT0000 complete [ 2574.816486] LustreError: 75018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.847447] LustreError: 75018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2576.005029] LustreError: 75002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443784 with bad export cookie 5512898319454389722 [ 2576.013236] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2576.016857] LustreError: 75002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2576.404616] Lustre: server umount lustre-MDT0001 complete [ 2588.164410] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2591.416718] Key type lgssc unregistered [ 2591.726571] LNet: 79894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.752337] LNetError: 79894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.779718] LNet: Removed LNI 192.168.204.135@tcp [ 2592.717735] Key type .llcrypt unregistered [ 2592.720580] Key type ._llcrypt unregistered [ 2606.661355] Key type ._llcrypt registered [ 2606.665645] Key type .llcrypt registered [ 2606.782305] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2607.139531] Key type .llcrypt unregistered [ 2607.143557] Key type ._llcrypt unregistered [ 2633.834892] Key type ._llcrypt registered [ 2633.848825] Key type .llcrypt registered [ 2633.985352] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2635.179936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2635.272897] alg: No test for adler32 (adler32-zlib) [ 2636.517927] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2636.784973] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2638.552658] Key type lgssc registered [ 2639.983436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.187928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.232717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.832177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.864608] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2659.369734] LustreError: 81800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.415446] LustreError: 81800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2664.153033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.516980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2669.125217] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2677.335800] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2686.208767] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2693.908517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.136506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2699.300571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2701.422918] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2710.299554] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2724.832496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.834864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.882775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.912869] Lustre: Skipped 1 previous similar message [ 2726.933382] Lustre: server umount lustre-OST0000 complete [ 2735.084559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.107825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.109175] Lustre: Skipped 2 previous similar messages [ 2735.166230] Lustre: Skipped 1 previous similar message [ 2740.196166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2740.200428] Lustre: Skipped 1 previous similar message [ 2744.289185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.297194] Lustre: Skipped 1 previous similar message [ 2745.311131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2745.558772] Lustre: server umount lustre-MDT0000 complete [ 2749.418331] LustreError: 82527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.437596] LustreError: 82527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2749.792967] LustreError: 84381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443958 with bad export cookie 3752443082587563884 [ 2749.809333] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.812707] LustreError: 84381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2750.227306] Lustre: server umount lustre-MDT0001 complete [ 2763.142658] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2766.502377] Key type lgssc unregistered [ 2766.855864] LNet: 85339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.862526] LNetError: 85339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2767.913898] LNet: Removed LNI 192.168.204.135@tcp [ 2769.149173] Key type .llcrypt unregistered [ 2769.152101] Key type ._llcrypt unregistered [ 2794.028636] Key type ._llcrypt registered [ 2794.031508] Key type .llcrypt registered [ 2794.134441] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2795.250917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2795.266382] alg: No test for adler32 (adler32-zlib) [ 2796.250944] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2796.483670] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2798.151319] Key type lgssc registered [ 2799.621583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2808.539607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2808.559609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.970124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2814.619193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2819.559515] LustreError: 86756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.582819] LustreError: 86756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2822.807259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.247652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2827.704763] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2835.327403] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2841.050470] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.680070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.941523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.381585] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2854.152714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2861.157095] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2869.529995] Lustre: server umount lustre-OST0000 complete [ 2879.461132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2879.471074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.483515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.183857] Lustre: server umount lustre-MDT0000 complete [ 2885.089754] Lustre: 86067:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444078/real 1788444078] req@000000008654546a x1875319461011840/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444094 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2885.097245] LustreError: 86750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.125763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.125772] Lustre: Skipped 1 previous similar message [ 2885.169181] LustreError: 86750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2887.933799] LustreError: 86736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444096 with bad export cookie 10127972949161051957 [ 2887.941699] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.950889] LustreError: 86736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2888.280631] Lustre: server umount lustre-MDT0001 complete [ 2899.723872] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2903.894308] Key type lgssc unregistered [ 2904.391627] LNet: 90304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2904.401358] LNetError: 90304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2904.422966] LNet: Removed LNI 192.168.204.135@tcp [ 2905.201083] Key type .llcrypt unregistered [ 2905.202969] Key type ._llcrypt unregistered [ 2928.699650] Key type ._llcrypt registered [ 2928.708081] Key type .llcrypt registered [ 2928.872930] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2929.614464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2929.794337] alg: No test for adler32 (adler32-zlib) [ 2930.822778] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2930.992568] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2932.615275] Key type lgssc registered [ 2933.865455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2941.222859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2941.239628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2942.689857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.789786] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2952.163166] LustreError: 91720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.183456] LustreError: 91720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2956.673639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2956.977857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.785928] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2967.894880] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2973.385627] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2981.023081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.437037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2987.598435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2989.485431] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2996.952709] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3007.972334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.973968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3007.975758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3007.990325] Lustre: Skipped 1 previous similar message [ 3013.093808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3013.108761] Lustre: Skipped 2 previous similar messages [ 3013.636644] Lustre: server umount lustre-OST0000 complete [ 3018.212373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3018.214298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3018.222595] Lustre: Skipped 1 previous similar message [ 3018.234377] Lustre: Skipped 1 previous similar message [ 3023.330678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.341335] Lustre: Skipped 1 previous similar message [ 3023.923062] Lustre: server umount lustre-MDT0000 complete [ 3027.706949] LustreError: 91700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444236 with bad export cookie 6122317331284444848 [ 3027.712381] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.737522] LustreError: 91700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3028.216675] Lustre: server umount lustre-MDT0001 complete [ 3040.319995] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3044.272356] Key type lgssc unregistered [ 3044.680192] LNet: 95259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3044.693336] LNetError: 95259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3044.735048] LNet: Removed LNI 192.168.204.135@tcp [ 3045.993211] Key type .llcrypt unregistered [ 3045.996769] Key type ._llcrypt unregistered [ 3072.758339] Key type ._llcrypt registered [ 3072.759994] Key type .llcrypt registered [ 3072.915306] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3073.696170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.752990] alg: No test for adler32 (adler32-zlib) [ 3075.025334] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3075.237309] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3076.919316] Key type lgssc registered [ 3078.184675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3086.026780] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3086.062793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3101.663780] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.783867] LustreError: 96647:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3133.471219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3140.075362] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3148.078857] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.330889] Lustre: server umount lustre-OST0000 complete [ 3167.688288] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:06:15 (1788444375) [ 3176.032541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.438160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3181.556505] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3181.579230] LustreError: 98287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3181.585351] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3186.668224] LustreError: 98286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.815282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.276223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.746409] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3205.359510] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3211.504317] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3219.179505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.742444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3221.825037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3226.543194] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3236.098439] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3251.685179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3251.692745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.709369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.718022] Lustre: Skipped 1 previous similar message [ 3254.566117] Lustre: server umount lustre-OST0000 complete [ 3261.922648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.924224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.935659] Lustre: Skipped 2 previous similar messages [ 3261.948796] Lustre: Skipped 1 previous similar message [ 3264.893057] Lustre: server umount lustre-MDT0000 complete [ 3267.043443] LustreError: 99240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3267.070644] LustreError: 99240:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3268.521497] LustreError: 98267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444477 with bad export cookie 4313478936772490710 [ 3268.528335] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3268.855939] Lustre: server umount lustre-MDT0001 complete [ 3282.021206] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3284.719604] Key type lgssc unregistered [ 3284.993926] LNet: 101982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3285.006678] LNetError: 101982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3285.022265] LNet: Removed LNI 192.168.204.135@tcp [ 3285.990722] Key type .llcrypt unregistered [ 3285.992499] Key type ._llcrypt unregistered [ 3329.572698] Key type ._llcrypt registered [ 3329.575462] Key type .llcrypt registered [ 3329.774320] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3331.392736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.485527] alg: No test for adler32 (adler32-zlib) [ 3332.604643] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3332.996656] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3334.754135] Key type lgssc registered [ 3336.022915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3345.619328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3345.643192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.227944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.350840] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3357.669175] LustreError: 104168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.685342] LustreError: 104168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3362.025136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.585422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.020398] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3376.949471] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3383.229214] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3390.523222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.878694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.960162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3397.682104] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3405.610288] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3417.575423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.589616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.589780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.599861] Lustre: Skipped 1 previous similar message [ 3417.640523] Lustre: Skipped 1 previous similar message [ 3421.382079] Lustre: server umount lustre-OST0000 complete [ 3427.825942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.835404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.837820] Lustre: Skipped 1 previous similar message [ 3427.860316] Lustre: Skipped 1 previous similar message [ 3432.211316] Lustre: server umount lustre-MDT0000 complete [ 3432.933848] LustreError: 104167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.907824] LustreError: 104148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444645 with bad export cookie 17270379712619678832 [ 3436.917247] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3436.932162] LustreError: 104148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3437.525806] Lustre: server umount lustre-MDT0001 complete [ 3447.598191] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3450.417198] Key type lgssc unregistered [ 3450.699506] LNet: 107706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3450.718773] LNetError: 107706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3450.739189] LNet: Removed LNI 192.168.204.135@tcp [ 3451.852944] Key type .llcrypt unregistered [ 3451.856823] Key type ._llcrypt unregistered [ 3474.177917] Key type ._llcrypt registered [ 3474.180519] Key type .llcrypt registered [ 3474.334872] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3476.454086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.584458] alg: No test for adler32 (adler32-zlib) [ 3477.729651] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3478.101908] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3479.839204] Key type lgssc registered [ 3481.110488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3488.762427] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3488.783574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3504.351401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3509.471654] LustreError: 108952:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3536.159414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3542.808372] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3550.997425] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3557.920381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.349490] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3558.365954] LustreError: 109976:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3558.373761] LustreError: 109976:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3558.380080] LustreError: 109976:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3558.482813] Lustre: server umount lustre-MDT0000 complete [ 3558.488501] LustreError: 109976:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3579.359202] Lustre: 108964:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444772/real 1788444772] req@000000004a22782b x1875320176119168/t0(0) o101->MGC192.168.204.135@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788444788 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3579.401421] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.419560] LustreError: 108964:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3579.661782] Lustre: server umount lustre-OST0000 complete [ 3589.401261] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3598.468251] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3637.955468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3638.349892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3638.431422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3638.578864] Lustre: lustre-MDT0000: new disk, initializing [ 3638.731163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.744617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3645.072911] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3656.046155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.116575] Lustre: 112768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3656.223676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3656.232280] Lustre: Skipped 1 previous similar message [ 3656.360506] Lustre: lustre-MDT0001: new disk, initializing [ 3656.549102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3656.615986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3656.626071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3660.697251] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3670.628756] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3676.766910] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3682.992258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3683.306852] Lustre: lustre-OST0000: new disk, initializing [ 3683.312701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3683.319946] Lustre: 114738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3683.431716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.104936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3690.116848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3690.264991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3690.883533] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3701.806903] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3707.494444] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3707.678977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3712.458316] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3712.652659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3716.066688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3716.068297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.074951] LustreError: Skipped 1 previous similar message [ 3716.086279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3716.095892] Lustre: Skipped 1 previous similar message [ 3716.129137] Lustre: Skipped 1 previous similar message [ 3721.186680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3721.200582] Lustre: Skipped 1 previous similar message [ 3726.316103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.345521] Lustre: Skipped 1 previous similar message [ 3728.863168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3729.168361] Lustre: server umount lustre-OST0000 complete [ 3736.547665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3736.575550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.586400] Lustre: Skipped 1 previous similar message [ 3740.091858] Lustre: server umount lustre-MDT0000 complete [ 3744.399406] LustreError: 114737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444953 with bad export cookie 8044427183222845493 [ 3744.414050] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.417476] LustreError: 114737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3744.817114] Lustre: server umount lustre-MDT0001 complete [ 3755.128620] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:16:02 (1788444962) [ 3760.483536] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3781.135109] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3792.480945] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3830.777664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3831.121609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3831.153541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3831.276991] Lustre: lustre-MDT0000: new disk, initializing [ 3831.442743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.467259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3836.393222] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3848.350331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.498488] Lustre: 119645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3848.539681] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3848.543484] Lustre: Skipped 1 previous similar message [ 3848.633908] Lustre: lustre-MDT0001: new disk, initializing [ 3848.761846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3848.801306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3848.815847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3854.399273] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3865.358996] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3871.781596] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3879.527269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.816897] Lustre: lustre-OST0000: new disk, initializing [ 3879.820099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3879.825579] Lustre: 121612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3879.906268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.407732] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3886.727326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3886.757901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3886.952285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3896.355596] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3901.458317] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3901.705896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3906.566974] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3906.866950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3912.673051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3912.680190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.685311] LustreError: Skipped 1 previous similar message [ 3912.688424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.727540] Lustre: Skipped 2 previous similar messages [ 3915.560775] Lustre: server umount lustre-OST0000 complete [ 3920.355699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3920.370795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.919136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.926923] Lustre: Skipped 2 previous similar messages [ 3934.175156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3934.436383] Lustre: server umount lustre-MDT0000 complete [ 3937.249371] LustreError: 120835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.284027] LustreError: 120835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3938.710871] LustreError: 120438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445147 with bad export cookie 8044427183222846375 [ 3938.726876] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3938.734834] LustreError: 120438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3939.020610] Lustre: server umount lustre-MDT0001 complete [ 3948.622420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.325704] LustreError: 123830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.488721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3954.294929] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3954.655967] LustreError: 123831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3959.779461] LustreError: 123830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3963.146849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3968.068105] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3975.329043] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3981.137811] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3988.919777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3995.717714] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4002.604100] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4012.181251] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4019.174098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.174510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4019.200840] Lustre: Skipped 1 previous similar message [ 4019.208239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4019.246732] Lustre: Skipped 5 previous similar messages [ 4022.592865] Lustre: server umount lustre-OST0000 complete [ 4029.411792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.447472] Lustre: Skipped 1 previous similar message [ 4034.033568] Lustre: server umount lustre-MDT0000 complete [ 4034.534479] LustreError: 123831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4034.557323] LustreError: 123831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4038.441088] LustreError: 123812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445247 with bad export cookie 8044427183222847257 [ 4038.445123] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4038.450932] LustreError: 123812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4038.863816] Lustre: server umount lustre-MDT0001 complete [ 4050.799023] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 4054.248346] Key type lgssc unregistered [ 4054.602760] LNet: 127467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4054.614644] LNetError: 127467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4054.628154] LNet: Removed LNI 192.168.204.135@tcp [ 4055.671882] Key type .llcrypt unregistered [ 4055.673599] Key type ._llcrypt unregistered [ 4074.451078] Key type ._llcrypt registered [ 4074.454766] Key type .llcrypt registered [ 4074.590577] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 4087.639228] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4089.253345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4089.268894] alg: No test for adler32 (adler32-zlib) [ 4090.227277] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4090.503034] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 4092.255183] Key type lgssc registered [ 4093.859736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4130.765308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4130.835356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.237642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4132.274222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4132.461920] Lustre: lustre-MDT0000: new disk, initializing [ 4132.714209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.797505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4138.914632] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4149.418801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4149.489470] Lustre: 130612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4149.527364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4149.531419] Lustre: Skipped 1 previous similar message [ 4149.594454] Lustre: lustre-MDT0001: new disk, initializing [ 4149.716644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4149.764251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4149.775830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4154.880528] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4164.853509] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4171.479748] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.463752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.899329] Lustre: lustre-OST0000: new disk, initializing [ 4178.906026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4178.918108] Lustre: 132581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4179.123171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.409613] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4188.302745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4188.323534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4188.495981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4198.789700] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4204.162699] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4204.400199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4209.486969] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4209.819852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4214.250263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4214.253473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.256488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.301110] Lustre: Skipped 1 previous similar message [ 4218.142448] Lustre: server umount lustre-OST0000 complete [ 4224.484214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.496134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.505055] Lustre: Skipped 2 previous similar messages [ 4229.603478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.721574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4234.725154] Lustre: Skipped 1 previous similar message [ 4236.255943] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4236.541669] Lustre: server umount lustre-MDT0000 complete [ 4241.726812] LustreError: 130606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445450 with bad export cookie 16029974032390637705 [ 4241.728740] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4241.739447] LustreError: 130606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4242.304870] Lustre: server umount lustre-MDT0001 complete [ 4254.879078] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:24:22 (1788445462) [ 4264.414258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.213377] LustreError: 134998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4265.250039] LustreError: 134998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4265.347756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.721515] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4270.567088] LustreError: 134999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4275.687410] LustreError: 134998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4278.394325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.933218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.170244] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4292.396521] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4298.841332] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4303.140681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.163292] Lustre: Skipped 1 previous similar message [ 4303.183797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4303.191949] Lustre: Skipped 1 previous similar message [ 4307.459545] Lustre: server umount lustre-MDT0000 complete [ 4308.449143] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.475518] LustreError: 134994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4311.272467] LustreError: 134978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445520 with bad export cookie 16029974032390638594 [ 4311.285926] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4311.743137] Lustre: server umount lustre-MDT0001 complete [ 4321.354743] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:25:28 (1788445528) [ 4329.202307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.929508] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4350.047478] LustreError: 137413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4375.775993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4382.489844] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4389.466589] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4391.828387] Lustre: server umount lustre-OST0000 complete [ 4400.371785] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:26:48 (1788445608) [ 4406.491495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4407.042214] LustreError: 138858:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.159697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.803971] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4417.506841] LustreError: 138859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.529287] LustreError: 138859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4424.285600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.863636] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4439.601051] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4445.484329] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4453.678851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.151363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.163199] Lustre: Skipped 1 previous similar message [ 4461.132290] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4470.284733] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4488.162034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.177103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4488.184400] Lustre: Skipped 3 previous similar messages [ 4488.217106] Lustre: Skipped 2 previous similar messages [ 4494.085180] Lustre: server umount lustre-MDT0000 complete [ 4498.412749] LustreError: 138855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4498.435405] LustreError: 138855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4498.768182] LustreError: 141457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445707 with bad export cookie 16029974032390639259 [ 4498.773099] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.785054] LustreError: 141457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4499.223730] Lustre: server umount lustre-MDT0001 complete [ 4513.825042] Lustre: server umount lustre-OST0000 complete [ 4523.629716] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:28:51 (1788445731) [ 4531.407631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.950352] LustreError: 142531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.036780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.460238] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4546.868208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.224832] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4560.638629] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4566.873403] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4573.956892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.444518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4581.232587] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4589.286758] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4594.308709] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4594.602107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4599.766703] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4600.032731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4602.340121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4602.351794] LustreError: Skipped 1 previous similar message [ 4602.359447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.374734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4602.382073] Lustre: Skipped 3 previous similar messages [ 4603.360157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4608.374827] Lustre: server umount lustre-OST0000 complete [ 4613.600545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.620834] Lustre: Skipped 2 previous similar messages [ 4619.107486] Lustre: server umount lustre-MDT0000 complete [ 4623.229443] LustreError: 142511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445832 with bad export cookie 16029974032390640323 [ 4623.231696] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4623.243967] LustreError: 142511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4623.733897] Lustre: server umount lustre-MDT0001 complete [ 4634.107538] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:30:41 (1788445841) [ 4642.370797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4657.951625] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4663.071690] LustreError: 146693:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4688.800610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.807814] Lustre: Skipped 2 previous similar messages [ 4695.617981] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4702.959629] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.940561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.410975] LustreError: 146718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4710.423461] LustreError: 146718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4710.533500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4714.524874] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4723.709073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.383105] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4739.723921] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4746.990568] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4751.648425] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4751.841419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4756.268969] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4756.451607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4760.037510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.042268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4760.056939] Lustre: Skipped 2 previous similar messages [ 4760.057466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4760.063233] Lustre: Skipped 8 previous similar messages [ 4764.596531] Lustre: server umount lustre-MDT0000 complete [ 4768.375163] LustreError: 146700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445977 with bad export cookie 16029974032390641352 [ 4768.385469] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4768.394710] LustreError: 146700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4768.750717] Lustre: server umount lustre-MDT0001 complete [ 4782.389503] Lustre: server umount lustre-OST0000 complete [ 4790.519567] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:33:18 (1788445998) [ 4797.137405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4812.832872] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4817.951527] LustreError: 150889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4855.008639] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4865.678142] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4872.738238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.066769] LustreError: 150914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.105982] LustreError: 150914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4873.211932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4877.733890] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4886.861384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.498130] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4897.605184] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4903.207562] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4910.838265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4911.068596] Lustre: lustre-OST0001: new disk, initializing [ 4911.072249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4911.077377] Lustre: 153765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4919.448283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4919.482769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4919.600394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4919.682794] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4930.557437] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4934.630448] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4934.881325] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4939.122747] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4939.375190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4943.334382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4943.349699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.366801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4947.759305] Lustre: server umount lustre-OST0000 complete [ 4955.616612] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4955.629201] LustreError: Skipped 2 previous similar messages [ 4957.814591] Lustre: server umount lustre-OST0001 complete [ 4976.095180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4976.361060] Lustre: server umount lustre-MDT0000 complete [ 4979.859381] LustreError: 153764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446188 with bad export cookie 16029974032390642304 [ 4979.873607] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4979.873868] LustreError: 153764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4980.162701] Lustre: server umount lustre-MDT0001 complete [ 5003.929841] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:36:51 (1788446211) [ 5005.303539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5007.079349] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5008.765830] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:36:56 (1788446216) [ 5015.889573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.323535] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5016.386235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5016.678424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.693484] Lustre: Skipped 6 previous similar messages [ 5021.982866] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5031.761577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.947363] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5031.985820] Lustre: 157327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5037.404134] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5044.938419] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5050.789503] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5057.482988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5057.690286] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5064.516395] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5065.343061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5072.214561] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5077.258423] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5077.504953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5082.437536] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5082.694768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5085.676140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5085.696495] LustreError: Skipped 1 previous similar message [ 5085.707716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.740266] Lustre: Skipped 5 previous similar messages [ 5085.753863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.758205] Lustre: Skipped 12 previous similar messages [ 5090.928723] Lustre: server umount lustre-OST0000 complete [ 5112.762673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5119.175748] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5126.441340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.455575] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5126.607411] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5131.731123] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5131.977457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5136.679361] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.697839] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5136.784536] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5136.969322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5142.623469] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff935320e2a000.ost_server_uuid 50 [ 5144.206388] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff935320e2a000.ost_server_uuid in FULL state after 0 sec [ 5144.364361] LustreError: 158745:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.35@tcp) returned error from blocking AST (req@00000000637fe623 x1875320818114176 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d2010e15/0xde75e873b76163da lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0xed7c95298ab08147 expref: 5 pid: 158745 timeout: 5244 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5144.431701] LustreError: lustre-MDT0000: A client on nid 192.168.204.35@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5144.447493] LustreError: 157324:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.35@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d2010e15/0xde75e873b76163da lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0xed7c95298ab08147 expref: 6 pid: 158745 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5150.186664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5150.201622] LustreError: Skipped 1 previous similar message [ 5154.828805] Lustre: server umount lustre-OST0000 complete [ 5164.596220] Lustre: server umount lustre-MDT0000 complete [ 5167.595870] LustreError: 157334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5167.627920] LustreError: 157334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5168.084967] LustreError: 157320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446377 with bad export cookie 16029974032390643487 [ 5168.097435] LustreError: 157320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5168.105612] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5174.553924] Lustre: server umount lustre-MDT0001 complete [ 5187.059573] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5190.200957] Key type lgssc unregistered [ 5190.541809] LNet: 163441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5190.552033] LNetError: 163441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5190.573762] LNet: Removed LNI 192.168.204.135@tcp [ 5191.557140] Key type .llcrypt unregistered [ 5191.558725] Key type ._llcrypt unregistered [ 5214.538727] Key type ._llcrypt registered [ 5214.541647] Key type .llcrypt registered [ 5214.661241] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5215.732372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5215.754384] alg: No test for adler32 (adler32-zlib) [ 5216.778800] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5217.036241] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5218.743204] Key type lgssc registered [ 5219.695189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5226.858271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5226.873110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.262747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.515225] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5238.754364] LustreError: 164855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.771193] LustreError: 164855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5239.864163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.272460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5244.273158] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5251.101088] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5256.683067] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5263.877616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5264.270716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5270.625191] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5273.548073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5278.453721] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5283.592467] Lustre: Failing over lustre-MDT0000 [ 5283.821436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.840635] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.853900] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5283.991905] Lustre: server umount lustre-MDT0000 complete [ 5285.046097] LustreError: 164850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.930084] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.968295] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5294.052944] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5294.076755] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5296.063865] LustreError: 164837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.35@tcp arrived at 1788446505 with bad export cookie 3590308079702936556 [ 5296.078086] LustreError: 164837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5303.137734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.253468] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.510711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5303.598628] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5307.803557] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5308.903227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5308.912047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5310.096855] Lustre: lustre-MDT0000: Denying connection for new client 532e6613-5411-4207-afb3-c1e726903ec5 (at 192.168.204.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5315.199626] Lustre: lustre-MDT0000: Denying connection for new client 532e6613-5411-4207-afb3-c1e726903ec5 (at 192.168.204.35@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5319.137281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.138688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5319.148431] Lustre: Skipped 3 previous similar messages [ 5319.169663] Lustre: Skipped 1 previous similar message [ 5324.264686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5324.277560] Lustre: Skipped 1 previous similar message [ 5324.529445] Lustre: server umount lustre-OST0000 complete [ 5328.474172] LustreError: 168584:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5328.485427] Lustre: 167934:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5328.494508] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5328.519410] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5328.540711] LustreError: 167934:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f8e8ff20 x1875321998584448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5328.554446] LustreError: 167934:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5328.565195] LustreError: 167934:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5328.577948] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5329.040544] Lustre: server umount lustre-MDT0000 complete [ 5329.388502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.417116] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.438752] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5333.276378] LustreError: 164837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446542 with bad export cookie 3590308079702936654 [ 5333.284707] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.295326] LustreError: 164837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5335.645748] Lustre: server umount lustre-MDT0001 complete [ 5345.940826] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5348.699246] Key type lgssc unregistered [ 5348.968750] LNet: 169339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5348.985945] LNetError: 169339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5350.055867] LNet: Removed LNI 192.168.204.135@tcp [ 5350.927423] Key type .llcrypt unregistered [ 5350.929462] Key type ._llcrypt unregistered [ 5376.711073] Key type ._llcrypt registered [ 5376.718848] Key type .llcrypt registered [ 5376.857286] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5378.836663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5378.861661] alg: No test for adler32 (adler32-zlib) [ 5380.065179] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5380.396417] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5382.119454] Key type lgssc registered [ 5383.261686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5390.663630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5390.698430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.170069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5396.005830] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5402.593908] LustreError: 170756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.605206] LustreError: 170756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5403.056806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.368130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5406.919554] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5413.272911] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5418.960589] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5424.697354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5424.957243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.770626] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5437.236067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5437.639367] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5444.581398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5444.588258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.606613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5447.664879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5447.665352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5447.681154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.771791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5452.777823] Lustre: Skipped 1 previous similar message [ 5457.898222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5457.908089] Lustre: Skipped 1 previous similar message [ 5458.400161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5458.614516] Lustre: server umount lustre-OST0000 complete [ 5463.009986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.027411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5463.034549] Lustre: Skipped 2 previous similar messages [ 5473.248990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5473.259654] Lustre: Skipped 3 previous similar messages [ 5476.319285] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5476.564294] Lustre: server umount lustre-MDT0000 complete [ 5478.369088] LustreError: 170756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5478.398664] LustreError: 170756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5480.482807] LustreError: 170737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446689 with bad export cookie 3716715019401437086 [ 5480.487882] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5480.498343] LustreError: 170737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5480.862520] Lustre: server umount lustre-MDT0001 complete [ 5492.021854] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5494.978315] Key type lgssc unregistered [ 5495.321246] LNet: 174302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5495.334666] LNetError: 174302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5495.359634] LNet: Removed LNI 192.168.204.135@tcp [ 5496.174654] Key type .llcrypt unregistered [ 5496.176865] Key type ._llcrypt unregistered [ 5527.695653] Key type ._llcrypt registered [ 5527.699453] Key type .llcrypt registered [ 5527.842603] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5528.732908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5528.787539] alg: No test for adler32 (adler32-zlib) [ 5529.797495] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5530.078067] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5531.863270] Key type lgssc registered [ 5532.936212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5540.995485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5541.044319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.537486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.224435] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5551.012590] LustreError: 176005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.032648] LustreError: 176005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5553.067849] LustreError: 176004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.352542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.738841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.747699] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5566.245809] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5572.032458] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5578.730617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5579.092256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.277850] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5588.333949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5591.946597] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5600.742309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.745225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.768172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.723981] Lustre: server umount lustre-OST0000 complete [ 5606.880315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5606.887709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.898817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.931438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5608.934207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.049412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5614.057581] Lustre: Skipped 2 previous similar messages [ 5615.226676] Lustre: server umount lustre-MDT0000 complete [ 5618.155965] LustreError: 177059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.174842] LustreError: 177059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5618.937569] LustreError: 175984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446827 with bad export cookie 16064741823761059011 [ 5618.956217] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5619.426164] Lustre: server umount lustre-MDT0001 complete [ 5630.153908] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5632.919900] Key type lgssc unregistered [ 5633.232758] LNet: 179541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5633.239988] LNetError: 179541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5633.265992] LNet: Removed LNI 192.168.204.135@tcp [ 5634.035031] Key type .llcrypt unregistered [ 5634.038049] Key type ._llcrypt unregistered [ 5655.728339] Key type ._llcrypt registered [ 5655.729888] Key type .llcrypt registered [ 5655.849194] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5657.090374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.098445] alg: No test for adler32 (adler32-zlib) [ 5658.126325] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5658.414118] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5660.080638] Key type lgssc registered [ 5660.924508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5670.350180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5670.366460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.680958] Lustre: *** cfs_fail_loc=135, val=0*** [ 5671.762762] LustreError: 181062:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5671.768557] LustreError: 181062:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 5671.780222] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5671.780222] [ 5671.789674] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5671.803838] LustreError: 181024:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5671.813784] LustreError: 181024:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5671.829139] LustreError: 181024:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5671.840293] LustreError: 181024:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5671.945562] Lustre: server umount lustre-MDT0000 complete [ 5671.949856] LustreError: 181024:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5677.729871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.012688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.032602] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5683.177875] LustreError: 181433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.202988] LustreError: 181433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5687.266066] LustreError: 181434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.117606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5689.393348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5692.754332] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5699.081832] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5703.611586] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5707.752460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.755051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.766572] Lustre: Skipped 1 previous similar message [ 5707.780380] Lustre: Skipped 1 previous similar message [ 5711.600377] Lustre: server umount lustre-MDT0000 complete [ 5712.866798] LustreError: 181433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.889135] LustreError: 181433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5714.912186] LustreError: 181414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446923 with bad export cookie 8546265057039343035 [ 5714.915924] LustreError: 181434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.935721] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5715.247557] Lustre: server umount lustre-MDT0001 complete [ 5725.055780] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5727.224629] Key type lgssc unregistered [ 5727.510738] LNet: 183758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5727.517857] LNetError: 183758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5727.533782] LNet: Removed LNI 192.168.204.135@tcp [ 5728.425750] Key type .llcrypt unregistered [ 5728.427037] Key type ._llcrypt unregistered [ 5748.370374] Key type ._llcrypt registered [ 5748.375679] Key type .llcrypt registered [ 5748.530729] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5748.945319] Key type .llcrypt unregistered [ 5748.949582] Key type ._llcrypt unregistered [ 5765.171424] Key type ._llcrypt registered [ 5765.175211] Key type .llcrypt registered [ 5765.334461] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5766.352708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.377475] alg: No test for adler32 (adler32-zlib) [ 5767.369604] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5767.585568] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5769.240155] Key type lgssc registered [ 5770.201889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5776.929772] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5776.955588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5792.607435] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5797.730056] LustreError: 185894:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 5824.415761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5830.559609] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5837.560815] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5843.696845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.061476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5844.127233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5847.932501] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5849.058661] LustreError: 185919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.069127] LustreError: 185919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5854.185460] LustreError: 185920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.816502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.142790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5860.132712] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5866.441771] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5871.864819] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5874.402313] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5886.946481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5886.954991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.980629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.023898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5890.029764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5890.036233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.940812] Lustre: server umount lustre-OST0000 complete [ 5897.188304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5897.202463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.226845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.237528] Lustre: Skipped 1 previous similar message [ 5900.257318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5900.262772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5900.278045] Lustre: Skipped 1 previous similar message [ 5902.232997] Lustre: server umount lustre-MDT0000 complete [ 5905.377407] LustreError: 186932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5905.401514] LustreError: 186932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5906.206307] LustreError: 188556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447115 with bad export cookie 457307688119313074 [ 5906.223928] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5906.231736] LustreError: 188556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5906.686708] Lustre: server umount lustre-MDT0001 complete [ 5917.667586] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5920.673888] Key type lgssc unregistered [ 5920.935447] LNet: 189910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5920.945553] LNetError: 189910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5920.965563] LNet: Removed LNI 192.168.204.135@tcp [ 5921.865800] Key type .llcrypt unregistered [ 5921.867400] Key type ._llcrypt unregistered [ 5944.667979] Key type ._llcrypt registered [ 5944.670225] Key type .llcrypt registered [ 5944.801557] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5945.619224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5945.706040] alg: No test for adler32 (adler32-zlib) [ 5946.750379] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5947.044551] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5948.687183] Key type lgssc registered [ 5949.933872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5957.604256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5957.636390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.137500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5963.833293] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5969.383424] LustreError: 191323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.394188] LustreError: 191323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5972.876452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.249453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5977.884655] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5985.456816] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5991.460253] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5998.584612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5998.990610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6004.153035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6005.731178] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6013.532128] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6019.290880] Lustre: Failing over lustre-MDT0000 [ 6019.556466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.563810] LustreError: 191322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.586188] Lustre: Skipped 1 previous similar message [ 6019.611797] LustreError: 191322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6019.681183] Lustre: server umount lustre-MDT0000 complete [ 6021.266101] LustreError: 191319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.674710] LustreError: 192050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.695046] LustreError: 192050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6029.793825] LustreError: 191319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.813335] LustreError: 191319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6038.016123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.094585] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6038.302103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.378810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6041.731437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6042.236485] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6043.625936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6043.639616] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6043.684311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6046.161689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6053.872424] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6059.489932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6059.496466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6059.508030] LustreError: Skipped 1 previous similar message [ 6059.534805] Lustre: Skipped 1 previous similar message [ 6059.547469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6061.537770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6061.554777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.571744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.108672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6064.115136] Lustre: Skipped 1 previous similar message [ 6065.224983] Lustre: server umount lustre-OST0000 complete [ 6069.220040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.220321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6069.229975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.241103] Lustre: Skipped 1 previous similar message [ 6074.337616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6074.342697] Lustre: Skipped 1 previous similar message [ 6075.317153] Lustre: server umount lustre-MDT0000 complete [ 6078.606463] LustreError: 193953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447287 with bad export cookie 4846496742644803569 [ 6078.607408] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6078.613915] LustreError: 193953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6078.904399] Lustre: server umount lustre-MDT0001 complete [ 6089.167900] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6091.827973] Key type lgssc unregistered [ 6092.119658] LNet: 196214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6092.128025] LNetError: 196214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6092.145462] LNet: Removed LNI 192.168.204.135@tcp [ 6092.867467] Key type .llcrypt unregistered [ 6092.871740] Key type ._llcrypt unregistered [ 6112.820476] Key type ._llcrypt registered [ 6112.822818] Key type .llcrypt registered [ 6112.893223] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6113.475240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6113.524704] alg: No test for adler32 (adler32-zlib) [ 6114.613295] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6114.888337] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6116.599219] Key type lgssc registered [ 6117.614729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.670483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.707661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.159561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.136536] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6136.291594] LustreError: 197607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.326922] LustreError: 197607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6136.748092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6137.105262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6141.261384] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6148.614897] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6153.053945] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6158.828648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6159.215778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6165.051943] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6170.979826] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6171.520120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6178.065609] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6189.493788] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6201.400267] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6212.579727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.582975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.589399] LustreError: Skipped 1 previous similar message [ 6212.602753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.524677] Lustre: server umount lustre-OST0000 complete [ 6222.821831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.824821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6222.838478] Lustre: Skipped 2 previous similar messages [ 6222.856040] Lustre: Skipped 2 previous similar messages [ 6225.859575] Lustre: server umount lustre-MDT0000 complete [ 6227.944373] LustreError: 199448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6227.977779] LustreError: 199448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6229.685607] LustreError: 197586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447438 with bad export cookie 5723598074340507077 [ 6229.686771] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.711913] LustreError: 197586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6230.357048] Lustre: server umount lustre-MDT0001 complete [ 6242.785080] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6246.345693] Key type lgssc unregistered [ 6246.682835] LNet: 201484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6246.690386] LNetError: 201484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6246.721087] LNet: Removed LNI 192.168.204.135@tcp [ 6247.721264] Key type .llcrypt unregistered [ 6247.723584] Key type ._llcrypt unregistered [ 6267.157430] Key type ._llcrypt registered [ 6267.159211] Key type .llcrypt registered [ 6267.316149] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6268.838627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6268.858141] alg: No test for adler32 (adler32-zlib) [ 6269.995465] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6270.316259] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6272.063673] Key type lgssc registered [ 6273.424335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6282.312877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6282.336619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.819652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6287.518403] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6293.986182] LustreError: 202752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.010636] LustreError: 202752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6296.318896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.717508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6301.812183] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6311.403222] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6317.152253] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6324.105957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6324.637757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.759610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6330.358789] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6336.994952] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6342.521180] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6351.380674] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6362.192729] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6369.177441] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6380.005517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6380.013134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.030379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.072563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.080410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.086511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.526617] Lustre: server umount lustre-OST0000 complete [ 6393.314292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.323186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6393.329144] Lustre: Skipped 1 previous similar message [ 6393.335321] Lustre: Skipped 1 previous similar message [ 6395.258937] Lustre: server umount lustre-MDT0000 complete [ 6398.433959] LustreError: 205822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.456169] LustreError: 205822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6398.574870] LustreError: 202731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447607 with bad export cookie 16956587973960726980 [ 6398.577499] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.584799] LustreError: 202731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6398.882278] Lustre: server umount lustre-MDT0001 complete [ 6409.234694] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6411.532616] Key type lgssc unregistered [ 6411.823966] LNet: 207728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6411.831909] LNetError: 207728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6411.846932] LNet: Removed LNI 192.168.204.135@tcp [ 6412.620887] Key type .llcrypt unregistered [ 6412.629675] Key type ._llcrypt unregistered [ 6437.123567] Key type ._llcrypt registered [ 6437.129181] Key type .llcrypt registered [ 6437.288860] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6438.523103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6438.743239] alg: No test for adler32 (adler32-zlib) [ 6439.841319] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6440.078637] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6441.719363] Key type lgssc registered [ 6443.324905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6451.105517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6451.119504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6452.865569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.209446] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6462.499725] LustreError: 209146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6462.520971] LustreError: 209146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6466.211268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6466.722352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6471.458776] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6479.738669] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6485.631874] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.215031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.517973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6497.477473] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6497.662655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6503.579861] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6517.224994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6517.228088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6517.241085] Lustre: Skipped 1 previous similar message [ 6517.251831] Lustre: Skipped 1 previous similar message [ 6520.967947] Lustre: server umount lustre-OST0000 complete [ 6527.457723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.462169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6527.468809] Lustre: Skipped 1 previous similar message [ 6527.479953] Lustre: Skipped 1 previous similar message [ 6531.214045] Lustre: server umount lustre-MDT0000 complete [ 6532.577937] LustreError: 210165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.598297] LustreError: 210165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6534.646473] LustreError: 209128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447743 with bad export cookie 4307069168333363367 [ 6534.663329] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.174980] Lustre: server umount lustre-MDT0001 complete [ 6544.448667] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6547.132695] Key type lgssc unregistered [ 6547.470704] LNet: 212843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6547.477474] LNetError: 212843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6547.497740] LNet: Removed LNI 192.168.204.135@tcp [ 6548.516911] Key type .llcrypt unregistered [ 6548.520066] Key type ._llcrypt unregistered [ 6566.634418] Key type ._llcrypt registered [ 6566.637794] Key type .llcrypt registered [ 6566.764711] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6567.760897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6567.852243] alg: No test for adler32 (adler32-zlib) [ 6568.858517] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6569.122083] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6570.834365] Key type lgssc registered [ 6571.727485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6577.495362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6577.511462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.998676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.342841] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6588.386438] LustreError: 214110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6588.415418] LustreError: 214110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6593.505776] LustreError: 214111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6593.951602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.571604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6600.174597] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6607.261581] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.121110] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6619.829744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6620.274743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.353261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6627.498813] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6634.100044] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.316715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.323078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.335428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.336404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.342153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.353551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.380433] Lustre: Skipped 1 previous similar message [ 6653.570735] Lustre: server umount lustre-OST0000 complete [ 6660.577402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.596691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.696942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.702917] Lustre: Skipped 2 previous similar messages [ 6670.823889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6670.832721] Lustre: Skipped 1 previous similar message [ 6671.327092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6671.461215] Lustre: server umount lustre-MDT0000 complete [ 6674.641052] LustreError: 214093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447883 with bad export cookie 511058378018963629 [ 6674.648204] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6674.656969] LustreError: 214093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6674.918242] Lustre: server umount lustre-MDT0001 complete [ 6684.545684] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6686.987563] Key type lgssc unregistered [ 6687.262754] LNet: 217798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6687.269099] LNetError: 217798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6687.293595] LNet: Removed LNI 192.168.204.135@tcp [ 6688.065517] Key type .llcrypt unregistered [ 6688.067333] Key type ._llcrypt unregistered [ 6706.540048] Key type ._llcrypt registered [ 6706.545127] Key type .llcrypt registered [ 6706.673861] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6708.047657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6708.073412] alg: No test for adler32 (adler32-zlib) [ 6709.070364] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6709.457563] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6711.231208] Key type lgssc registered [ 6712.075915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6719.518749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6719.535609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6720.848623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.580708] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6730.216250] LustreError: 219067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.234117] LustreError: 219067:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6735.063668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6735.330679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6735.518229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.541838] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6747.563826] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6752.801168] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6759.138237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6759.438765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.261289] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6765.631030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6775.087296] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6786.523223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6786.664536] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6787.145722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6794.005371] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6801.523455] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.050480] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6814.910578] Lustre: Permanently deactivating lustre-OST0001 [ 6814.926162] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6820.925562] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6826.235543] Lustre: Permanently reactivating lustre-OST0001 [ 6826.257834] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6826.263363] Lustre: Skipped 2 previous similar messages [ 6830.686419] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.704590] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6830.711639] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6830.725406] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6830.776919] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6841.315233] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6841.321108] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.321927] LustreError: Skipped 1 previous similar message [ 6841.322877] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6841.351926] Lustre: Skipped 2 previous similar messages [ 6841.740916] Lustre: server umount lustre-OST0001 complete [ 6842.342193] LustreError: 220888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6842.361239] LustreError: 220888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6847.459974] LustreError: 220889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6847.464620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.465281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6847.480581] LustreError: 220889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6847.521670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.527529] Lustre: Skipped 2 previous similar messages [ 6852.581283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6852.589268] LustreError: 223185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6852.595608] Lustre: Skipped 1 previous similar message [ 6852.630186] LustreError: 223185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6857.696893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6857.699670] LustreError: 220889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.707122] Lustre: Skipped 1 previous similar message [ 6857.739808] LustreError: 220889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6859.233165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6859.418086] Lustre: server umount lustre-OST0000 complete [ 6862.829717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.841364] Lustre: Skipped 1 previous similar message [ 6867.940050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.957246] Lustre: Skipped 3 previous similar messages [ 6868.720424] Lustre: server umount lustre-MDT0000 complete [ 6872.191485] LustreError: 219046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448081 with bad export cookie 4711964864948837201 [ 6872.199422] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6872.204328] LustreError: 219046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6872.592298] Lustre: server umount lustre-MDT0001 complete [ 6881.579534] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6884.178118] Key type lgssc unregistered [ 6884.441889] LNet: 224141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6884.450905] LNetError: 224141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6884.473113] LNet: Removed LNI 192.168.204.135@tcp [ 6885.296107] Key type .llcrypt unregistered [ 6885.298038] Key type ._llcrypt unregistered [ 6923.943537] Key type ._llcrypt registered [ 6923.947076] Key type .llcrypt registered [ 6924.055845] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6924.975890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6925.004622] alg: No test for adler32 (adler32-zlib) [ 6926.151135] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6926.364803] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6928.023377] Key type lgssc registered [ 6929.154425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6936.853852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.867769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6938.038234] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6938.051810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6938.152189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.265628] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6950.685692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.798120] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6950.835256] Lustre: 226347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6951.035418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6958.613788] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6972.404850] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6980.162620] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6986.049053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6986.254090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6986.492570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6993.096797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6993.282799] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7000.180881] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7003.733095] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7010.408951] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7017.875816] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7026.573930] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7036.515371] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7052.999975] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7053.004773] Lustre: Skipped 1 previous similar message [ 7087.236856] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7087.239493] Lustre: Skipped 4 previous similar messages [ 7155.622349] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7155.631479] Lustre: Skipped 8 previous similar messages [ 7172.579503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7172.581410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7172.593458] LustreError: Skipped 1 previous similar message [ 7172.595095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.613604] Lustre: Skipped 1 previous similar message [ 7172.628820] Lustre: Skipped 1 previous similar message [ 7176.776911] Lustre: server umount lustre-OST0000 complete [ 7181.288290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7181.305330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.317605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.818484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7182.819050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.676724] Lustre: server umount lustre-MDT0000 complete [ 7190.428669] LustreError: 229414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448399 with bad export cookie 18115091757015287115 [ 7190.444596] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7190.448610] LustreError: 229414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7190.723699] Lustre: server umount lustre-MDT0001 complete [ 7204.726133] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7207.970688] Key type lgssc unregistered [ 7208.349954] LNet: 231133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7208.358738] LNetError: 231133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7208.379562] LNet: Removed LNI 192.168.204.135@tcp [ 7209.653746] Key type .llcrypt unregistered [ 7209.656413] Key type ._llcrypt unregistered [ 7234.466802] Key type ._llcrypt registered [ 7234.472944] Key type .llcrypt registered [ 7234.644477] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7236.174259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7236.202641] alg: No test for adler32 (adler32-zlib) [ 7237.295696] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7237.679075] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7239.439543] Key type lgssc registered [ 7240.621760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7248.153331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7248.164669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.572406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7253.925371] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7259.111911] LustreError: 232545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.123946] LustreError: 232545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7261.992745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.554712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.230259] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7274.937627] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7279.792374] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7285.927714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.279375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7292.147044] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 7294.445922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7299.642329] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7323.105123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7323.110572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7323.118760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.155869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.162455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.162729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.204091] Lustre: Skipped 1 previous similar message [ 7328.465328] Lustre: server umount lustre-OST0000 complete [ 7334.377299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.394340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.415294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.774302] Lustre: server umount lustre-MDT0000 complete [ 7340.514220] LustreError: 232545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.539879] LustreError: 232545:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7343.006676] LustreError: 232528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448551 with bad export cookie 11184053545649156010 [ 7343.013606] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7343.482826] Lustre: server umount lustre-MDT0001 complete [ 7354.359134] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7357.904986] Key type lgssc unregistered [ 7358.192681] LNet: 236240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7358.210928] LNetError: 236240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7358.237375] LNet: Removed LNI 192.168.204.135@tcp [ 7359.360357] Key type .llcrypt unregistered [ 7359.362618] Key type ._llcrypt unregistered [ 7385.747851] Key type ._llcrypt registered [ 7385.749632] Key type .llcrypt registered [ 7385.870740] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7388.043204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7388.108188] alg: No test for adler32 (adler32-zlib) [ 7389.267461] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7389.625333] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7391.295437] Key type lgssc registered [ 7392.571023] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7665.364715] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7668.445714] Key type lgssc unregistered [ 7668.876194] LNet: 238266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7668.890682] LNetError: 238266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7668.905635] LNet: Removed LNI 192.168.204.135@tcp [ 7669.961653] Key type .llcrypt unregistered [ 7669.967959] Key type ._llcrypt unregistered [ 7699.777473] Key type ._llcrypt registered [ 7699.782305] Key type .llcrypt registered [ 7699.995480] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7718.746438] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7719.583990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7719.619292] alg: No test for adler32 (adler32-zlib) [ 7720.705478] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7720.910778] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7722.567172] Key type lgssc registered [ 7723.520318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7754.779214] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7474 sec ========== 11:22:42 (1788448962) [ 7756.244486] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:22:43 (1788448963) === [ 7759.388174] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:22:47 (1788448967) === [ 7799.897641] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7802.736603] Key type lgssc unregistered [ 7802.996072] LNet: 243507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7803.007976] LNetError: 243507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7803.026874] LNet: Removed LNI 192.168.204.135@tcp [ 7803.958685] Key type .llcrypt unregistered [ 7803.963784] Key type ._llcrypt unregistered