[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492326067 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003283] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008563] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.012143] LSM: Security Framework initializing [ 0.013056] Yama: becoming mindful. [ 0.014041] SELinux: Initializing. [ 0.015071] *** VALIDATE selinux *** [ 0.022793] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027028] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030132] *** VALIDATE tmpfs *** [ 0.032258] *** VALIDATE proc *** [ 0.033208] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036213] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042761] debug: unmapping init [mem 0xffffffffa5659000-0xffffffffa5660fff] [ 0.044214] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046032] ... version: 2 [ 0.047019] ... bit width: 48 [ 0.048018] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051021] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055491] smp: Bringing up secondary CPUs ... [ 0.056546] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.060618] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.259026] node 0 deferred pages initialised in 194ms [ 0.263018] devtmpfs: initialized [ 0.264362] x86/mm: Memory block size: 128MB [ 0.267379] gcov: version magic: 0x41383552 [ 0.269547] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.273110] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275432] pinctrl core: initialized pinctrl subsystem [ 0.278260] [ 0.278837] ************************************************************* [ 0.281019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283019] ** ** [ 0.285021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.288025] ** ** [ 0.290015] ** This means that this kernel is built to expose internal ** [ 0.292018] ** IOMMU data structures, which may compromise security on ** [ 0.293017] ** your system. ** [ 0.296021] ** ** [ 0.298017] ** If you see this message and you are not debugging the ** [ 0.300022] ** kernel, report this immediately to your vendor! ** [ 0.303040] ** ** [ 0.305017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.307015] ************************************************************* [ 0.309948] NET: Registered protocol family 16 [ 0.312621] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.315080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320146] cpuidle: using governor menu [ 0.321831] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.324597] PCI: Using configuration type 1 for base access [ 0.325242] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.337081] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.339121] cryptd: max_cpu_qlen set to 1000 [ 0.341174] ACPI: Added _OSI(Module Device) [ 0.343026] ACPI: Added _OSI(Processor Device) [ 0.345020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.352846] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360647] ACPI: Interpreter enabled [ 0.362095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.364044] ACPI: Using IOAPIC for interrupt routing [ 0.365298] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.369474] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.380507] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.382049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.385029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.389106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.394538] acpiphp: Slot [2] registered [ 0.396196] acpiphp: Slot [5] registered [ 0.397135] acpiphp: Slot [6] registered [ 0.399180] acpiphp: Slot [7] registered [ 0.400192] acpiphp: Slot [8] registered [ 0.402189] acpiphp: Slot [9] registered [ 0.403135] acpiphp: Slot [10] registered [ 0.404170] acpiphp: Slot [3] registered [ 0.406129] acpiphp: Slot [4] registered [ 0.407104] acpiphp: Slot [11] registered [ 0.409125] acpiphp: Slot [12] registered [ 0.410150] acpiphp: Slot [13] registered [ 0.412128] acpiphp: Slot [14] registered [ 0.413127] acpiphp: Slot [15] registered [ 0.415120] acpiphp: Slot [16] registered [ 0.417119] acpiphp: Slot [17] registered [ 0.418124] acpiphp: Slot [18] registered [ 0.420147] acpiphp: Slot [19] registered [ 0.421155] acpiphp: Slot [20] registered [ 0.423134] acpiphp: Slot [21] registered [ 0.424123] acpiphp: Slot [22] registered [ 0.426118] acpiphp: Slot [23] registered [ 0.427149] acpiphp: Slot [24] registered [ 0.428124] acpiphp: Slot [25] registered [ 0.430152] acpiphp: Slot [26] registered [ 0.431095] acpiphp: Slot [27] registered [ 0.432113] acpiphp: Slot [28] registered [ 0.434115] acpiphp: Slot [29] registered [ 0.435120] acpiphp: Slot [30] registered [ 0.436118] acpiphp: Slot [31] registered [ 0.438066] PCI host bridge to bus 0000:00 [ 0.439027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.441031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444051] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.447059] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.449034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.452046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.454182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.457000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460356] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.470021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.481020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.483957] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.485562] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.488052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.489762] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508416] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.518023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.524012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.543066] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.556559] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.582020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.591436] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626144] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.635018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.643022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.658022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.665934] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.671018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.688014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.696779] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.701021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.704017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.720020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.733298] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.737546] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.739503] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.742411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.745271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.751038] iommu: Default domain type: Passthrough [ 0.752845] SCSI subsystem initialized [ 0.754182] ACPI: bus type USB registered [ 0.756204] usbcore: registered new interface driver usbfs [ 0.759129] usbcore: registered new interface driver hub [ 0.761423] usbcore: registered new device driver usb [ 0.763314] pps_core: LinuxPPS API ver. 1 registered [ 0.766019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.769083] PTP clock support registered [ 0.771214] EDAC MC: Ver: 3.0.0 [ 0.774179] PCI: Using ACPI for IRQ routing [ 0.777041] NetLabel: Initializing [ 0.778013] NetLabel: domain hash size = 128 [ 0.779012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.781091] NetLabel: unlabeled traffic allowed by default [ 0.783136] vgaarb: loaded [ 0.784298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.791909] clocksource: Switched to clocksource kvm-clock [ 0.888985] VFS: Disk quotas dquot_6.6.0 [ 0.890713] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892653] *** VALIDATE ramfs *** [ 0.893657] *** VALIDATE hugetlbfs *** [ 0.894820] pnp: PnP ACPI init [ 0.896686] pnp: PnP ACPI: found 6 devices [ 0.912960] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916572] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919353] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921937] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925607] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928084] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931034] NET: Registered protocol family 2 [ 0.933843] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940229] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943835] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.948719] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.951663] TCP: Hash tables configured (established 65536 bind 65536) [ 0.955389] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.958529] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.961789] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.965034] NET: Registered protocol family 1 [ 0.967574] RPC: Registered named UNIX socket transport module. [ 0.970044] RPC: Registered udp transport module. [ 0.972414] RPC: Registered tcp transport module. [ 0.974898] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.977794] NET: Registered protocol family 44 [ 0.979565] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.981857] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984111] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.986969] PCI: CLS 0 bytes, default 64 [ 0.988642] Unpacking initramfs... [ 2.377220] debug: unmapping init [mem 0xffffa07ffcc54000-0xffffa07ffffbffff] [ 2.380787] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.383186] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.386372] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.919594] Initialise system trusted keyrings [ 2.922816] Key type blacklist registered [ 2.924603] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.934973] zbud: loaded [ 2.938519] *** VALIDATE nfs *** [ 2.939950] *** VALIDATE nfs4 *** [ 2.941733] pstore: using deflate compression [ 2.945763] Platform Keyring initialized [ 3.082241] NET: Registered protocol family 38 [ 3.084112] Key type asymmetric registered [ 3.085671] Asymmetric key parser 'x509' registered [ 3.087653] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.091075] io scheduler mq-deadline registered [ 3.092920] io scheduler kyber registered [ 3.094822] io scheduler bfq registered [ 3.096804] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.099979] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.104324] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.107442] ACPI: Power Button [PWRF] [ 3.112809] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.123129] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.135738] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.142640] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.156733] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.184672] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.212124] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.216606] Non-volatile memory driver v1.3 [ 3.218271] Linux agpgart interface v0.103 [ 3.245401] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.247498] vda: detected capacity change from 0 to 76771328 [ 3.263888] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.267112] vdb: detected capacity change from 0 to 1073741824 [ 3.281848] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.285190] vdc: detected capacity change from 0 to 2621440000 [ 3.302415] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.306221] vdd: detected capacity change from 0 to 2621440000 [ 3.329215] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331780] vde: detected capacity change from 0 to 4294967296 [ 3.348912] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351760] vdf: detected capacity change from 0 to 4294967296 [ 3.358102] libphy: Fixed MDIO Bus: probed [ 3.370501] usbcore: registered new interface driver usbserial_generic [ 3.372768] usbserial: USB Serial support registered for generic [ 3.375170] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.379468] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.380973] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.382852] mousedev: PS/2 mouse device common for all mice [ 3.385098] rtc_cmos 00:05: RTC can wake from S4 [ 3.386237] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.389785] rtc_cmos 00:05: registered as rtc0 [ 3.393562] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.396785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.397484] intel_pstate: CPU model not supported [ 3.407145] hid: raw HID events driver (C) Jiri Kosina [ 3.408948] usbcore: registered new interface driver usbhid [ 3.410554] usbhid: USB HID core driver [ 3.412120] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.413079] drop_monitor: Initializing network drop monitor service [ 3.419763] Initializing XFRM netlink socket [ 3.422499] NET: Registered protocol family 10 [ 3.426744] Segment Routing with IPv6 [ 3.428445] NET: Registered protocol family 17 [ 3.430805] mpls_gso: MPLS GSO support [ 3.435651] RAS: Correctable Errors collector initialized. [ 3.437907] AVX version of gcm_enc/dec engaged. [ 3.439668] AES CTR mode by8 optimization enabled [ 3.516264] sched_clock: Marking stable (3516198251, 0)->(4483875295, -967677044) [ 3.520595] registered taskstats version 1 [ 3.522895] Loading compiled-in X.509 certificates [ 3.524713] zswap: loaded using pool lzo/zbud [ 3.549697] Key type big_key registered [ 3.563082] Key type encrypted registered [ 3.564564] ima: No TPM chip found, activating TPM-bypass! [ 3.568159] ima: Allocated hash algorithm: sha1 [ 3.570409] ima: No architecture policies found [ 3.572315] evm: Initialising EVM extended attributes: [ 3.574274] evm: security.selinux [ 3.575419] evm: security.ima [ 3.576516] evm: security.capability [ 3.577921] evm: HMAC attrs: 0x1 [ 3.580680] rtc_cmos 00:05: setting system clock to 2026-09-07 15:57:57 UTC (1788796677) [ 3.586772] debug: unmapping init [mem 0xffffffffa6603000-0xffffffffa67fffff] [ 3.589651] debug: unmapping init [mem 0xffffffffa5382000-0xffffffffa5658fff] [ 3.598095] Write protecting the kernel read-only data: 28672k [ 3.600721] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 3.603206] debug: unmapping init [mem 0xffffffffa4314000-0xffffffffa43fffff] [ 3.638198] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.645230] systemd[1]: Detected virtualization kvm. [ 3.647539] systemd[1]: Detected architecture x86-64. [ 3.649556] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.679788] systemd[1]: No hostname configured. [ 3.681495] systemd[1]: Set hostname to . [ 3.683367] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685642] systemd[1]: Initializing machine ID from random generator. [ 3.740961] random: ln: uninitialized urandom read (6 bytes read) [ 3.861686] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864325] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.867273] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.874094] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.650866] device-mapper: uevent: version 1.0.3 [ 4.654647] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK [0[ 5.272345] random: fast init done m] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.522415] virtio_net virtio0 ens2: renamed from eth0 [ 5.624360] scsi host0: ata_piix [ 5.647993] scsi host1: ata_piix [ 5.649189] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.650908] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.014746] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.084671] random: crng init done [ 10.086151] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.689335] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.981089] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.221021] SELinux: Disabled at runtime. [ 12.278796] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.288081] systemd[1]: Detected virtualization kvm. [ 12.290023] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.776323] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.779531] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.784275] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.787223] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.789648] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.797864] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.802952] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ 12.879735] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.258827] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.540401] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.608087] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.721045] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.735722] EDAC sbridge: Ver: 1.1.2 [ 15.169306] Key type dns_resolver registered [ 15.471536] NFS: Registering the id_resolver key type [ 15.473735] Key type id_resolver registered [ 15.475286] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg108-server login: [ 32.357097] hrtimer: interrupt took 4033516 ns [ 66.500886] libcfs: loading out-of-tree module taints kernel. [ 66.575168] Key type ._llcrypt registered [ 66.581394] Key type .llcrypt registered [ 66.722440] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 87.175652] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 89.164192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.183148] alg: No test for adler32 (adler32-zlib) [ 90.904595] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 91.636235] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 93.360231] Key type lgssc registered [ 95.265801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.989565] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 163.669266] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 178.939679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 178.972196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.304655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 180.360709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 180.489668] Lustre: lustre-MDT0000: new disk, initializing [ 180.551892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.564450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.761941] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.458265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.619392] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.723275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.729041] Lustre: Skipped 1 previous similar message [ 202.903348] Lustre: lustre-MDT0001: new disk, initializing [ 203.013037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.108028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.121785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.660209] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.763975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.608932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.074888] Lustre: lustre-OST0000: new disk, initializing [ 225.092419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.110212] Lustre: 8477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.306454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.455463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.476941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.573048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.697531] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.240319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.479862] Lustre: lustre-OST0001: new disk, initializing [ 250.487665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 250.500899] Lustre: 9553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.601075] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.123890] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.775684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.805381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.958317] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.491572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 281.851388] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 289.380887] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.844731] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing yml_node [ 302.448656] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 305.404562] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 308.927778] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 311.703943] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 12:03:03 EDT 2026 [ 334.247131] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 335.974636] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 353.252702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.254531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.270515] Lustre: Skipped 1 previous similar message [ 353.289050] Lustre: Skipped 3 previous similar messages [ 358.369380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.375129] Lustre: Skipped 1 previous similar message [ 359.166372] Lustre: server umount lustre-MDT0000 complete [ 363.342974] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797037 with bad export cookie 17606753045404714671 [ 363.344625] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.359538] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 363.492487] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.502077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.507942] Lustre: Skipped 3 previous similar messages [ 363.516450] Lustre: Skipped 2 previous similar messages [ 368.615623] LustreError: 6534:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 368.621205] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 368.636792] LustreError: 6534:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 368.643569] Lustre: Skipped 2 previous similar messages [ 369.785380] Lustre: server umount lustre-MDT0001 complete [ 374.327702] Lustre: server umount lustre-OST0000 complete [ 379.569779] Lustre: server umount lustre-OST0001 complete [ 387.638748] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 400.951692] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 443.979440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.369849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 444.429370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 444.562070] Lustre: lustre-MDT0000: new disk, initializing [ 444.659880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.695278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 449.958664] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 462.819845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.936747] Lustre: 15110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 462.982919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 462.986289] Lustre: Skipped 1 previous similar message [ 463.041856] Lustre: lustre-MDT0001: new disk, initializing [ 463.153820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 463.196435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 463.214547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.135606] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 478.886922] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 486.129300] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 494.538674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.999419] Lustre: lustre-OST0000: new disk, initializing [ 495.003110] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 495.017640] Lustre: 17078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 495.125126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.772328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 496.789795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 496.923672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 503.887269] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 515.099294] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.119870] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.492274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.998185] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 526.178900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 532.971799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.975747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.978685] LustreError: Skipped 1 previous similar message [ 532.984781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 533.001620] Lustre: Skipped 1 previous similar message [ 533.018642] Lustre: Skipped 1 previous similar message [ 534.700463] Lustre: server umount lustre-OST0000 complete [ 540.129521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 540.137203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.202219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 543.222537] Lustre: Skipped 1 previous similar message [ 545.910160] Lustre: server umount lustre-MDT0000 complete [ 548.321915] LustreError: 15121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.347739] LustreError: 15121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 550.189064] LustreError: 15102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797224 with bad export cookie 17606753045404716204 [ 550.213863] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.512822] Lustre: server umount lustre-MDT0001 complete [ 554.335618] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 12:07:06 (1788797226) [ 561.877370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.324678] LustreError: 19345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.390297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.025856] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 567.785071] LustreError: 19346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.903267] LustreError: 19345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.481199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.662558] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 590.687220] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 598.520909] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 607.456629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 607.728299] LustreError: 21165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.621631] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 624.289874] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 633.826667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 633.836834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.856896] Lustre: Skipped 1 previous similar message [ 633.865485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.874823] Lustre: Skipped 1 previous similar message [ 636.898658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 638.132702] Lustre: server umount lustre-OST0000 complete [ 643.554917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.576530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.608811] Lustre: Skipped 1 previous similar message [ 649.145254] Lustre: server umount lustre-MDT0000 complete [ 652.258363] LustreError: 19346:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.280649] LustreError: 19346:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 653.632437] LustreError: 19326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797327 with bad export cookie 17606753045404717086 [ 653.641597] LustreError: 19326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 653.650518] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 653.983604] Lustre: server umount lustre-MDT0001 complete [ 667.741228] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 671.002690] Key type lgssc unregistered [ 671.492376] LNet: 22881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.512747] LNetError: 22881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.546401] LNet: Removed LNI 192.168.201.108@tcp [ 672.714158] Key type .llcrypt unregistered [ 672.715977] Key type ._llcrypt unregistered [ 700.880817] Key type ._llcrypt registered [ 700.885360] Key type .llcrypt registered [ 700.998754] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 702.508706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.544179] alg: No test for adler32 (adler32-zlib) [ 703.572898] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 703.841631] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 705.526755] Key type lgssc registered [ 706.687236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 713.948979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.967518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.613717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 721.573938] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 725.986777] LustreError: 24290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.006611] LustreError: 24290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 730.617192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.900234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 735.721524] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 744.696327] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 751.694378] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 760.162892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 760.727094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.808706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 768.312687] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 778.506097] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 797.669534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.675921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.687376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 801.761296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 801.766132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 801.772864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.814172] Lustre: Skipped 1 previous similar message [ 804.186485] Lustre: server umount lustre-OST0000 complete [ 812.001870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.031269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.049727] Lustre: Skipped 1 previous similar message [ 815.457865] Lustre: server umount lustre-MDT0000 complete [ 817.128788] LustreError: 24290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.154018] LustreError: 24290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 819.309976] LustreError: 24270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797493 with bad export cookie 5789818023301869000 [ 819.317625] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.327574] LustreError: 24270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 819.888631] Lustre: server umount lustre-MDT0001 complete [ 832.193620] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 835.259871] Key type lgssc unregistered [ 835.772345] LNet: 28268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.795518] LNetError: 28268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.830826] LNet: Removed LNI 192.168.201.108@tcp [ 836.972959] Key type .llcrypt unregistered [ 836.988145] Key type ._llcrypt unregistered [ 868.043635] Key type ._llcrypt registered [ 868.047097] Key type .llcrypt registered [ 868.185541] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 870.006527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 870.050854] alg: No test for adler32 (adler32-zlib) [ 871.243127] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 871.553218] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 873.280621] Key type lgssc registered [ 874.432572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 883.653697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 883.666076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.166835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.555203] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 895.465663] LustreError: 29688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 895.499768] LustreError: 29688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 899.507825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.922556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 904.731168] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 912.471259] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 918.158335] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.005206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.725661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.553347] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 948.684982] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 955.183269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 965.601363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.611716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.614661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.654276] Lustre: Skipped 1 previous similar message [ 967.699292] Lustre: server umount lustre-OST0000 complete [ 971.750973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.764547] LustreError: Skipped 1 previous similar message [ 971.771642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.808144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 971.814978] Lustre: Skipped 1 previous similar message [ 975.847960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.852629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.056986] Lustre: server umount lustre-MDT0000 complete [ 980.961350] LustreError: 30695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.982165] LustreError: 30695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 981.981816] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797655 with bad export cookie 6138609648379113270 [ 982.002168] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.512282] Lustre: server umount lustre-MDT0001 complete [ 995.140836] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 998.519384] Key type lgssc unregistered [ 998.790562] LNet: 33587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.811707] LNetError: 33587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.847785] LNet: Removed LNI 192.168.201.108@tcp [ 1000.938346] Key type .llcrypt unregistered [ 1000.940197] Key type ._llcrypt unregistered [ 1026.372220] Key type ._llcrypt registered [ 1026.377799] Key type .llcrypt registered [ 1026.509192] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1028.342593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.413889] alg: No test for adler32 (adler32-zlib) [ 1029.626263] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1029.948954] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1031.657874] Key type lgssc registered [ 1033.322443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1042.203323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1042.246032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.912621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.497836] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1053.187565] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1053.211577] LustreError: 34998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1058.276260] LustreError: 34997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1058.854843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.281897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1064.700831] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1074.293842] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.941474] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1090.258282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.769463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.618329] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1107.114178] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.331480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1123.820336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1123.832584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.837289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1123.848715] Lustre: Skipped 1 previous similar message [ 1123.865272] Lustre: Skipped 1 previous similar message [ 1128.670890] Lustre: server umount lustre-OST0000 complete [ 1134.059459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.076221] Lustre: Skipped 1 previous similar message [ 1134.085871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1134.095666] Lustre: Skipped 1 previous similar message [ 1139.170563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1139.174561] Lustre: Skipped 1 previous similar message [ 1139.780442] Lustre: server umount lustre-MDT0000 complete [ 1143.268712] LustreError: 37608:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1143.300854] LustreError: 37608:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1143.901331] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797817 with bad export cookie 3235655814523385042 [ 1143.916949] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.937470] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1144.171944] Lustre: server umount lustre-MDT0001 complete [ 1157.475413] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1160.295607] Key type lgssc unregistered [ 1160.653717] LNet: 38561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.663372] LNetError: 38561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.683887] LNet: Removed LNI 192.168.201.108@tcp [ 1161.568916] Key type .llcrypt unregistered [ 1161.571277] Key type ._llcrypt unregistered [ 1188.667941] Key type ._llcrypt registered [ 1188.670699] Key type .llcrypt registered [ 1188.791602] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1190.148435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.232073] alg: No test for adler32 (adler32-zlib) [ 1191.286652] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1191.509845] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1193.170187] Key type lgssc registered [ 1194.260804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.112580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.128923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.648662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.929845] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1213.951087] LustreError: 39971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.968536] LustreError: 39971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1218.586033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.049996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.550610] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1232.144991] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1239.795919] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1246.810574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1247.212989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.241488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1253.489961] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1262.602486] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1270.766455] Lustre: lustre-OST0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1273.830686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.867412] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1273.873918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1273.923978] Lustre: Skipped 1 previous similar message [ 1275.847996] Lustre: lustre-OST0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 1276.341996] Lustre: server umount lustre-OST0000 complete [ 1298.402502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.432861] Lustre: Skipped 1 previous similar message [ 1298.449180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1298.455430] Lustre: Skipped 1 previous similar message [ 1303.522319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.624987] Lustre: server umount lustre-MDT0000 complete [ 1307.307914] LustreError: 40706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797981 with bad export cookie 11339852286577601787 [ 1307.316333] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1307.318631] LustreError: 40706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1313.767932] Lustre: server umount lustre-MDT0001 complete [ 1324.445891] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1326.981675] Key type lgssc unregistered [ 1327.259163] LNet: 43679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.265046] LNetError: 43679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.295766] LNet: Removed LNI 192.168.201.108@tcp [ 1329.345246] Key type .llcrypt unregistered [ 1329.346979] Key type ._llcrypt unregistered [ 1353.120423] Key type ._llcrypt registered [ 1353.123332] Key type .llcrypt registered [ 1353.249833] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1354.544662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.617266] alg: No test for adler32 (adler32-zlib) [ 1355.766423] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1356.073811] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1357.760318] Key type lgssc registered [ 1359.101868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1368.352535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1368.388408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.103275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1374.922308] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1379.303742] LustreError: 45089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.318430] LustreError: 45089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1382.768978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.137492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1387.695456] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1395.552571] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1400.845913] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1407.167278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1407.454950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.618384] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1414.660064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1421.396711] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1429.036424] Lustre: server umount lustre-MDT0000 complete [ 1429.466137] LustreError: 47319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1429.485061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1429.499854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.241094] LustreError: 45068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798106 with bad export cookie 8308158873706586884 [ 1432.247174] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1432.249789] LustreError: 45068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1434.568637] LustreError: 46997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.598218] LustreError: 46997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1434.613341] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.8@tcp (stopping) [ 1435.111127] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.140445] Lustre: Skipped 2 previous similar messages [ 1435.161312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1437.153940] LustreError: 45084:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.154238] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1437.175099] LustreError: 45084:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1438.655458] Lustre: server umount lustre-MDT0001 complete [ 1467.872189] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1468.313283] Lustre: server umount lustre-OST0000 complete [ 1489.913585] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1493.886437] Key type lgssc unregistered [ 1494.111806] LNet: 48951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.118728] LNetError: 48951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.157728] LNet: Removed LNI 192.168.201.108@tcp [ 1496.652475] Key type .llcrypt unregistered [ 1496.655328] Key type ._llcrypt unregistered [ 1526.958411] Key type ._llcrypt registered [ 1526.959766] Key type .llcrypt registered [ 1527.098827] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1528.271480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1528.435406] alg: No test for adler32 (adler32-zlib) [ 1529.481655] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1529.767752] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1531.480314] Key type lgssc registered [ 1533.394638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1543.889846] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1543.923778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.584934] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1564.704554] LustreError: 50340:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 1591.392632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.629898] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1608.278747] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1776.882972] Lustre: server umount lustre-OST0000 complete [ 1794.745714] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1797.394458] Key type lgssc unregistered [ 1797.854218] LNet: 52018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.872021] LNetError: 52018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.958384] LNet: Removed LNI 192.168.201.108@tcp [ 1799.950680] Key type .llcrypt unregistered [ 1799.956709] Key type ._llcrypt unregistered [ 1823.382271] Key type ._llcrypt registered [ 1823.385219] Key type .llcrypt registered [ 1823.469802] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1824.691206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.768787] alg: No test for adler32 (adler32-zlib) [ 1825.834254] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1826.095233] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1827.720176] Key type lgssc registered [ 1828.802212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.672625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.690188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.185027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.101948] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1847.777099] LustreError: 53429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.809193] LustreError: 53429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1850.111662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.648344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1855.589041] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1863.517599] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1869.915798] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1876.889978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.277032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.492797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1884.869583] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 1895.511351] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1902.053341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1902.068124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.084841] Lustre: Skipped 1 previous similar message [ 1902.085162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1902.096687] Lustre: Skipped 1 previous similar message [ 1904.997654] Lustre: server umount lustre-OST0000 complete [ 1912.291039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1912.293049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1912.293537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1912.308205] Lustre: Skipped 1 previous similar message [ 1912.329333] Lustre: Skipped 1 previous similar message [ 1915.981803] Lustre: server umount lustre-MDT0000 complete [ 1917.417427] LustreError: 54157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.003674] LustreError: 53411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798593 with bad export cookie 8120249080119750514 [ 1920.017663] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.020325] LustreError: 53411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1920.599828] Lustre: server umount lustre-MDT0001 complete [ 1932.444879] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 1934.920705] Key type lgssc unregistered [ 1935.181758] LNet: 56961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.192099] LNetError: 56961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.208534] LNet: Removed LNI 192.168.201.108@tcp [ 1935.948558] Key type .llcrypt unregistered [ 1935.950421] Key type ._llcrypt unregistered [ 1959.875097] Key type ._llcrypt registered [ 1959.884450] Key type .llcrypt registered [ 1960.011218] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 1961.214067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.315151] alg: No test for adler32 (adler32-zlib) [ 1962.406118] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 1962.697201] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 1964.467388] Key type lgssc registered [ 1965.607718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.108612] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1973.127838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.705820] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1993.824392] LustreError: 58341:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 2020.512339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.274403] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2035.082441] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2041.775223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2042.295946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.365486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2046.875453] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2047.459357] LustreError: 58366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.478527] LustreError: 58366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2052.580908] LustreError: 58365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.654532] LustreError: 58366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.672475] LustreError: 58366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2057.025845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.639693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2062.028576] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2069.787179] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2075.497176] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2078.178681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.187073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.187343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.317175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2083.327815] Lustre: Skipped 2 previous similar messages [ 2084.063987] Lustre: server umount lustre-OST0000 complete [ 2108.385440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.397568] Lustre: Skipped 1 previous similar message [ 2108.403211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.786384] Lustre: server umount lustre-MDT0000 complete [ 2113.506364] LustreError: 59381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.520880] LustreError: 59381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2117.496682] LustreError: 58347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798791 with bad export cookie 787026176980081062 [ 2117.497591] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.514913] LustreError: 58347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2124.077674] Lustre: server umount lustre-MDT0001 complete [ 2140.274953] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2143.870975] Key type lgssc unregistered [ 2144.098430] LNet: 62088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.104781] LNetError: 62088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.123345] LNet: Removed LNI 192.168.201.108@tcp [ 2144.963564] Key type .llcrypt unregistered [ 2144.970150] Key type ._llcrypt unregistered [ 2169.352672] Key type ._llcrypt registered [ 2169.355310] Key type .llcrypt registered [ 2169.454701] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2170.477438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.576456] alg: No test for adler32 (adler32-zlib) [ 2171.736620] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2171.994363] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2173.704430] Key type lgssc registered [ 2175.162204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2183.592395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2183.620282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.065237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.533350] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2194.407574] LustreError: 63499:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.446196] LustreError: 63499:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2198.846806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.452847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.860456] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2212.205533] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2218.654615] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.388569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2227.864440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.962149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2235.824767] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2246.615905] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2265.061063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2265.074986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.077101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2265.111971] Lustre: Skipped 1 previous similar message [ 2270.180483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2270.186589] Lustre: Skipped 1 previous similar message [ 2270.676206] Lustre: server umount lustre-OST0000 complete [ 2275.309459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.317555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.333904] Lustre: Skipped 1 previous similar message [ 2275.364031] Lustre: Skipped 2 previous similar messages [ 2280.418866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.741954] Lustre: server umount lustre-MDT0000 complete [ 2284.956959] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798958 with bad export cookie 17408812490783695054 [ 2284.983215] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.237880] Lustre: server umount lustre-MDT0001 complete [ 2299.798311] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2302.541919] Key type lgssc unregistered [ 2302.946145] LNet: 67051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.964963] LNetError: 67051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.998244] LNet: Removed LNI 192.168.201.108@tcp [ 2303.941976] Key type .llcrypt unregistered [ 2303.944499] Key type ._llcrypt unregistered [ 2330.310740] Key type ._llcrypt registered [ 2330.313236] Key type .llcrypt registered [ 2330.419199] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2331.745663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.936656] alg: No test for adler32 (adler32-zlib) [ 2333.027647] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2333.226172] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2334.952234] Key type lgssc registered [ 2335.984341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2346.836129] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 12:36:58 (1788799018) [ 2355.732470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.749939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.505909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.627309] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2367.975927] LustreError: 68752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.005855] LustreError: 68752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2371.778113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2372.135668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2376.016635] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2384.007217] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.414316] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.743117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2399.000592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.107190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2406.605186] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2416.655090] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2424.459136] Lustre: Failing over lustre-MDT0000 [ 2424.810628] LustreError: 68747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.987567] Lustre: server umount lustre-MDT0000 complete [ 2426.865338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.874509] LustreError: 68752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.886750] Lustre: Skipped 2 previous similar messages [ 2426.908764] LustreError: 68752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2429.902893] LustreError: 71220:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.942705] LustreError: 71220:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2435.012809] LustreError: 68749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.092118] LustreError: 68749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2437.475501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.694797] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.953495] Lustre: *** cfs_fail_loc=135, val=0*** [ 2438.090538] LustreError: 71843:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2438.104799] LustreError: 71843:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.108@tcp: cfg command failed: rc = -2 [ 2438.120483] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2438.120483] [ 2438.139783] LustreError: MGC192.168.201.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2438.165507] LustreError: 71831:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2438.180889] LustreError: 71831:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2438.203025] LustreError: 71831:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2438.214306] LustreError: 71831:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2438.251516] Lustre: server umount lustre-MDT0000 complete [ 2438.256558] LustreError: 71831:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2446.271934] LustreError: 69799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.293606] LustreError: 69799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2447.290293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.357722] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.365288] LustreError: 68744:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2447.616557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.682925] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2450.387701] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2452.970685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2453.018785] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2453.102476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2453.820851] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2463.212254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.221290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.223308] Lustre: Skipped 1 previous similar message [ 2463.236752] Lustre: Skipped 1 previous similar message [ 2468.322799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2468.338647] Lustre: Skipped 1 previous similar message [ 2469.274939] Lustre: server umount lustre-OST0000 complete [ 2478.574271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.579419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.581227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2478.592211] Lustre: Skipped 1 previous similar message [ 2483.684095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.696694] Lustre: Skipped 2 previous similar messages [ 2488.289013] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2488.548096] Lustre: server umount lustre-MDT0000 complete [ 2488.811768] LustreError: 68753:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.828060] LustreError: 68753:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2492.978503] LustreError: 68733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799166 with bad export cookie 7461740405625071602 [ 2492.981823] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.982861] LustreError: 68733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2493.207782] Lustre: server umount lustre-MDT0001 complete [ 2505.228310] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2507.967852] Key type lgssc unregistered [ 2508.369982] LNet: 73623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2508.383873] LNetError: 73623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2508.418542] LNet: Removed LNI 192.168.201.108@tcp [ 2509.503353] Key type .llcrypt unregistered [ 2509.509658] Key type ._llcrypt unregistered [ 2535.940198] Key type ._llcrypt registered [ 2535.943637] Key type .llcrypt registered [ 2536.055550] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2537.351896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2537.551423] alg: No test for adler32 (adler32-zlib) [ 2538.604559] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2538.891442] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2540.577151] Key type lgssc registered [ 2542.016594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2549.797571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2549.812615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.230088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.092735] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2561.508586] LustreError: 75034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.530075] LustreError: 75034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2564.954346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.260526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2570.594669] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2580.306618] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2587.822378] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.901638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.379991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.362262] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2616.197455] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2621.944884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2628.316394] Lustre: Failing over lustre-MDT0000 [ 2628.889849] Lustre: server umount lustre-MDT0000 complete [ 2631.651838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2631.663932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.688713] LustreError: 75034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.170896] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.181115] Lustre: Skipped 1 previous similar message [ 2632.710799] LustreError: 77528:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.731199] LustreError: 77528:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2637.283516] LustreError: 75034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.311264] LustreError: 75034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2639.723079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.865944] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.076285] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2640.282029] LustreError: 78126:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2640.293115] LustreError: 78126:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.108@tcp: cfg command failed: rc = -9 [ 2640.312952] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2640.312952] [ 2640.331060] LustreError: MGC192.168.201.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2640.358931] LustreError: 78115:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2640.374521] LustreError: 78115:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2640.396641] LustreError: 78115:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2640.403240] LustreError: 78115:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2640.458368] Lustre: server umount lustre-MDT0000 complete [ 2640.461513] LustreError: 78115:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2642.419160] LustreError: 75035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.447893] LustreError: 75035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2648.010342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2648.186372] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.475876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2648.524108] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2649.071638] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2653.420468] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2653.689299] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2653.751537] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2653.788966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2661.858397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.865503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.886793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.906851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.908388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.026415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2669.046707] Lustre: Skipped 2 previous similar messages [ 2674.165325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2674.168724] Lustre: Skipped 1 previous similar message [ 2675.168162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2675.495021] Lustre: server umount lustre-OST0000 complete [ 2684.386682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2684.394778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.396624] Lustre: Skipped 1 previous similar message [ 2684.399358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2686.104378] Lustre: server umount lustre-MDT0000 complete [ 2689.513165] LustreError: 75030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2689.538899] LustreError: 75030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2689.892560] LustreError: 75016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799363 with bad export cookie 17115887308550496604 [ 2689.902978] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2689.907092] LustreError: 75016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2690.276337] Lustre: server umount lustre-MDT0001 complete [ 2702.297220] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2705.886972] Key type lgssc unregistered [ 2706.262901] LNet: 79906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.276702] LNetError: 79906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2707.307442] LNet: Removed LNI 192.168.201.108@tcp [ 2708.541784] Key type .llcrypt unregistered [ 2708.544099] Key type ._llcrypt unregistered [ 2721.223195] Key type ._llcrypt registered [ 2721.226706] Key type .llcrypt registered [ 2721.414581] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2721.734510] Key type .llcrypt unregistered [ 2721.736603] Key type ._llcrypt unregistered [ 2750.499247] Key type ._llcrypt registered [ 2750.501652] Key type .llcrypt registered [ 2750.650054] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2752.113525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.175302] alg: No test for adler32 (adler32-zlib) [ 2753.215373] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2753.409496] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2755.120342] Key type lgssc registered [ 2756.198514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2763.994955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2764.032860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.563208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2770.144789] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2776.036610] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.068710] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2778.480547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.848488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.591962] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2791.118209] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2797.144472] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2804.039329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2804.418060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.548100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2811.015995] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2817.493899] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2829.301691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.302287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 2829.314358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.319380] Lustre: Skipped 1 previous similar message [ 2829.344689] LustreError: Skipped 1 previous similar message [ 2834.406267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2834.414507] Lustre: Skipped 2 previous similar messages [ 2835.174885] Lustre: server umount lustre-OST0000 complete [ 2839.524404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.540306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.560660] Lustre: Skipped 1 previous similar message [ 2844.646868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2847.576933] Lustre: server umount lustre-MDT0000 complete [ 2849.761193] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.781373] LustreError: 81801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2851.511447] LustreError: 81782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799525 with bad export cookie 18258687350030049455 [ 2851.513590] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.516494] LustreError: 82532:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 2851.516554] LustreError: 82532:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 2851.522599] LustreError: 81782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2851.895083] Lustre: server umount lustre-MDT0001 complete [ 2863.068637] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 2866.133115] Key type lgssc unregistered [ 2866.490531] LNet: 85338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2866.507255] LNetError: 85338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2866.548493] LNet: Removed LNI 192.168.201.108@tcp [ 2867.679210] Key type .llcrypt unregistered [ 2867.682149] Key type ._llcrypt unregistered [ 2893.621935] Key type ._llcrypt registered [ 2893.624555] Key type .llcrypt registered [ 2893.727660] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 2895.225436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2895.353093] alg: No test for adler32 (adler32-zlib) [ 2896.367757] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 2896.568201] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 2898.232820] Key type lgssc registered [ 2899.468043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2908.182104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2908.194122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.755916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.656518] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2919.911384] LustreError: 86747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.940015] LustreError: 86747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2925.028806] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.366219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.979636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2930.645535] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2938.237401] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2944.719046] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2953.134289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2953.820240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.920622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2962.542844] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 2973.541955] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.836044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2980.841522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.852633] LustreError: Skipped 1 previous similar message [ 2980.858384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.869220] Lustre: Skipped 1 previous similar message [ 2980.909359] Lustre: Skipped 1 previous similar message [ 2985.957160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2985.960058] Lustre: Skipped 1 previous similar message [ 2986.482309] Lustre: server umount lustre-OST0000 complete [ 2991.074312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2991.085980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.194820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2996.208810] Lustre: Skipped 2 previous similar messages [ 2997.112783] Lustre: server umount lustre-MDT0000 complete [ 3001.314481] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.335676] LustreError: 86748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3001.373644] LustreError: 86728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799675 with bad export cookie 9511820181216684095 [ 3001.380496] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3001.389842] LustreError: 86728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3001.790724] Lustre: server umount lustre-MDT0001 complete [ 3012.765665] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3015.794968] Key type lgssc unregistered [ 3016.132846] LNet: 90280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.146654] LNetError: 90280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.172509] LNet: Removed LNI 192.168.201.108@tcp [ 3017.270116] Key type .llcrypt unregistered [ 3017.272226] Key type ._llcrypt unregistered [ 3042.773446] Key type ._llcrypt registered [ 3042.775760] Key type .llcrypt registered [ 3042.872138] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3043.935888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.008695] alg: No test for adler32 (adler32-zlib) [ 3045.658347] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 3046.216939] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3048.034768] Key type lgssc registered [ 3049.889448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3059.242762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3059.288065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.971132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.512973] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3070.444520] LustreError: 91690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.467873] LustreError: 91690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3074.278034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3074.587745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.843876] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3087.680111] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.079828] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.305423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3099.642831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.803329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3106.020076] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3113.785435] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3129.840800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3129.851831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.883565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3130.355693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3130.361911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.783749] Lustre: server umount lustre-OST0000 complete [ 3135.981775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.995047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3136.014131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3136.022068] Lustre: Skipped 2 previous similar messages [ 3140.578987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3140.580376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.701310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.722902] Lustre: Skipped 2 previous similar messages [ 3150.304185] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3150.713247] Lustre: server umount lustre-MDT0000 complete [ 3150.824311] LustreError: 91686:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.857567] LustreError: 91686:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3155.583713] LustreError: 91670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799829 with bad export cookie 9450865219630880782 [ 3155.585848] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.598258] LustreError: 91670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3156.040971] Lustre: server umount lustre-MDT0001 complete [ 3169.623654] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3173.158862] Key type lgssc unregistered [ 3173.487782] LNet: 95243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3173.493711] LNetError: 95243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3173.525614] LNet: Removed LNI 192.168.201.108@tcp [ 3174.534338] Key type .llcrypt unregistered [ 3174.541910] Key type ._llcrypt unregistered [ 3195.970273] Key type ._llcrypt registered [ 3195.974283] Key type .llcrypt registered [ 3196.124182] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3197.216131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3197.237300] alg: No test for adler32 (adler32-zlib) [ 3198.305666] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 3198.637090] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3200.320326] Key type lgssc registered [ 3201.637645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3209.298912] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3209.315824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3224.864401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3229.984340] LustreError: 96607:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 3256.672713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.814884] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3272.447494] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3278.996500] Lustre: server umount lustre-OST0000 complete [ 3288.096978] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 12:52:40 (1788799960) [ 3295.846833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.240770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.670691] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3301.351252] LustreError: 98249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.362387] LustreError: 98249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3306.475707] LustreError: 98250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.281261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.087238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3314.288244] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3321.285563] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3325.854495] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3331.939582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3332.481308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3339.096976] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3341.682281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3345.877206] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3357.158258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3357.161211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.178769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3357.202512] Lustre: Skipped 1 previous similar message [ 3357.217527] Lustre: Skipped 1 previous similar message [ 3362.277478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3362.290399] Lustre: Skipped 1 previous similar message [ 3363.085886] Lustre: server umount lustre-OST0000 complete [ 3366.370720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3366.381855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.407866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3367.407187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.517132] LustreError: 98245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.534855] LustreError: 98245:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3372.622846] Lustre: server umount lustre-MDT0000 complete [ 3376.578579] LustreError: 98230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800050 with bad export cookie 13954350932439143876 [ 3376.586188] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3376.592243] LustreError: 98230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3377.131389] Lustre: server umount lustre-MDT0001 complete [ 3387.610262] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3390.436639] Key type lgssc unregistered [ 3390.838241] LNet: 101939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3390.847982] LNetError: 101939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3390.861837] LNet: Removed LNI 192.168.201.108@tcp [ 3391.758856] Key type .llcrypt unregistered [ 3391.766355] Key type ._llcrypt unregistered [ 3434.528271] Key type ._llcrypt registered [ 3434.530436] Key type .llcrypt registered [ 3434.619053] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3435.982276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3436.046783] alg: No test for adler32 (adler32-zlib) [ 3437.173975] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 3437.485861] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3439.177167] Key type lgssc registered [ 3440.478663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.928730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.948948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.506772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3454.088721] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3460.080631] LustreError: 104123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.105574] LustreError: 104123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3462.942962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3463.386608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3468.274712] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3476.108893] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3481.224303] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3487.683623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.998364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.755187] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3494.147295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3502.195303] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3514.849347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3514.850180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.854429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3514.862514] LustreError: Skipped 1 previous similar message [ 3514.880353] Lustre: Skipped 1 previous similar message [ 3517.514749] Lustre: server umount lustre-OST0000 complete [ 3524.577327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3524.592053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3524.610911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3524.620669] Lustre: Skipped 1 previous similar message [ 3527.685577] Lustre: server umount lustre-MDT0000 complete [ 3530.209520] LustreError: 106621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.236724] LustreError: 106621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3531.366700] LustreError: 104103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800205 with bad export cookie 17434573027329312978 [ 3531.385874] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3531.700644] Lustre: server umount lustre-MDT0001 complete [ 3542.247327] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 3545.139803] Key type lgssc unregistered [ 3545.455892] LNet: 107655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3545.462422] LNetError: 107655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3545.489208] LNet: Removed LNI 192.168.201.108@tcp [ 3546.376967] Key type .llcrypt unregistered [ 3546.378920] Key type ._llcrypt unregistered [ 3564.440755] Key type ._llcrypt registered [ 3564.442358] Key type .llcrypt registered [ 3564.544671] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3566.162276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3566.194529] alg: No test for adler32 (adler32-zlib) [ 3567.315379] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 3567.635932] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 3569.360308] Key type lgssc registered [ 3570.847820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3578.327637] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3578.347282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.953334] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3599.072308] LustreError: 108894:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 3625.760217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.487351] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3641.297738] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.458387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.864221] LustreError: MGC192.168.201.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3648.874868] LustreError: 109918:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3648.883843] LustreError: 109918:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3648.892628] LustreError: 109918:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3649.049454] Lustre: server umount lustre-MDT0000 complete [ 3649.056191] LustreError: 109918:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3669.986323] Lustre: 108231:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788800327/real 1788800327] req@0000000096f8e379 x1875693001510272/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788800343 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3670.034590] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.063789] LustreError: 108906:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3670.219768] Lustre: server umount lustre-OST0000 complete [ 3680.126569] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 3691.644985] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3731.127833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.413640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3731.450960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3731.555190] Lustre: lustre-MDT0000: new disk, initializing [ 3731.649109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.669211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3737.012639] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3749.571479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.733672] Lustre: 112708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3749.803950] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3749.808322] Lustre: Skipped 1 previous similar message [ 3749.958846] Lustre: lustre-MDT0001: new disk, initializing [ 3750.034108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3750.070067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3750.090395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3755.176768] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3763.797788] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3769.537276] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3776.987558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3777.298164] Lustre: lustre-OST0000: new disk, initializing [ 3777.307893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3777.324152] Lustre: 114682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3777.459507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3783.323976] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3785.273793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3785.283790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3785.361783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3791.783709] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3796.541351] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3796.752139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3801.345333] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3801.564122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3806.181873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3806.184313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3806.206436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.227526] Lustre: Skipped 1 previous similar message [ 3809.383704] Lustre: server umount lustre-OST0000 complete [ 3816.423842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.427626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3816.440763] Lustre: Skipped 2 previous similar messages [ 3816.460886] Lustre: Skipped 1 previous similar message [ 3819.789304] Lustre: server umount lustre-MDT0000 complete [ 3823.586570] LustreError: 113505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800497 with bad export cookie 9255356215177473020 [ 3823.594191] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3823.604842] LustreError: 113505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3824.058981] Lustre: server umount lustre-MDT0001 complete [ 3832.033698] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 13:01:44 (1788800504) [ 3836.197109] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3853.072260] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 3862.894825] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 3899.837713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.047710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3900.075696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3900.157186] Lustre: lustre-MDT0000: new disk, initializing [ 3900.240384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.255988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3904.539626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3914.696111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.814952] Lustre: 119593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3914.881933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3914.889448] Lustre: Skipped 1 previous similar message [ 3914.985811] Lustre: lustre-MDT0001: new disk, initializing [ 3915.065546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.109473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3915.123263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.326883] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3929.200697] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3935.550197] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.529132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3944.314121] Lustre: lustre-OST0000: new disk, initializing [ 3944.324291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3944.341245] Lustre: 121560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3944.471184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3951.509912] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 3952.784191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3952.802260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3953.017323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3961.875353] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.556619] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3966.713894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3972.477149] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3972.715766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3978.724134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.727268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3978.756543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3981.669530] Lustre: server umount lustre-OST0000 complete [ 3986.915680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3986.919304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3986.947682] Lustre: Skipped 1 previous similar message [ 3986.961520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3986.977048] Lustre: Skipped 1 previous similar message [ 3992.624607] Lustre: server umount lustre-MDT0000 complete [ 3994.081982] LustreError: 119606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.093971] LustreError: 119606:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3996.024282] LustreError: 119587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800669 with bad export cookie 9255356215177473909 [ 3996.027980] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.036677] LustreError: 119587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3996.324668] Lustre: server umount lustre-MDT0001 complete [ 4003.858974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.365500] LustreError: 123778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4004.457190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.400738] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4009.447774] LustreError: 123779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.577905] LustreError: 123778:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4017.384828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.031314] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4031.125104] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4036.906161] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4043.979757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.811582] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4058.469562] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4068.716493] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4072.938262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.943973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.949669] Lustre: Skipped 1 previous similar message [ 4072.953169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.957059] Lustre: Skipped 2 previous similar messages [ 4079.221990] Lustre: server umount lustre-OST0000 complete [ 4083.178129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.196889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4083.199436] Lustre: Skipped 2 previous similar messages [ 4083.216715] Lustre: Skipped 4 previous similar messages [ 4088.800299] Lustre: server umount lustre-MDT0000 complete [ 4092.419083] LustreError: 123758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800766 with bad export cookie 9255356215177474784 [ 4092.448739] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4093.027488] Lustre: server umount lustre-MDT0001 complete [ 4103.785242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 4106.220408] Key type lgssc unregistered [ 4106.474231] LNet: 127404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4106.481275] LNetError: 127404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4106.504577] LNet: Removed LNI 192.168.201.108@tcp [ 4107.596220] Key type .llcrypt unregistered [ 4107.598588] Key type ._llcrypt unregistered [ 4128.145949] Key type ._llcrypt registered [ 4128.151426] Key type .llcrypt registered [ 4128.295335] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 4147.731622] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 4148.993911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.073170] alg: No test for adler32 (adler32-zlib) [ 4150.281677] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 4150.561351] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 4152.344229] Key type lgssc registered [ 4153.814171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4195.316191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4195.357063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.686188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4196.729936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4196.877314] Lustre: lustre-MDT0000: new disk, initializing [ 4197.106610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4197.184094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4202.316922] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4213.243590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.367732] Lustre: 130555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4213.404224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4213.407174] Lustre: Skipped 1 previous similar message [ 4213.490448] Lustre: lustre-MDT0001: new disk, initializing [ 4213.622601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4213.674142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4213.687733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4217.924634] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4227.251976] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4232.694275] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4239.337276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4239.574176] Lustre: lustre-OST0000: new disk, initializing [ 4239.577283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4239.582927] Lustre: 132525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4239.653946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.385641] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4249.214663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4249.240617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4249.417081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4255.866215] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4261.140391] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4261.347345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4266.638888] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4266.912466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4270.054748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4270.058396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.078798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4270.086639] Lustre: Skipped 1 previous similar message [ 4275.172140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4275.175381] Lustre: Skipped 2 previous similar messages [ 4280.293733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4280.306210] Lustre: Skipped 1 previous similar message [ 4283.377853] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4283.731545] Lustre: server umount lustre-OST0000 complete [ 4290.547551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.549588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.556888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4290.559558] Lustre: Skipped 1 previous similar message [ 4295.650417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4295.655696] Lustre: Skipped 2 previous similar messages [ 4302.305109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4302.464783] Lustre: server umount lustre-MDT0000 complete [ 4306.840554] LustreError: 130545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800980 with bad export cookie 8210181994358000516 [ 4306.845303] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4306.855532] LustreError: 130545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4307.469104] Lustre: server umount lustre-MDT0001 complete [ 4317.172967] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 13:09:48 (1788800988) [ 4324.827843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.259396] LustreError: 134940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.277346] LustreError: 134940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4325.336612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.958114] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4330.476422] LustreError: 134941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4335.596696] LustreError: 134940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.372676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.823406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4343.600179] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4351.634601] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4358.403660] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4360.674066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.694038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.696577] Lustre: Skipped 2 previous similar messages [ 4364.835131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.601242] Lustre: server umount lustre-MDT0000 complete [ 4369.889927] LustreError: 134935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4369.905585] LustreError: 134935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4371.766565] LustreError: 134922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801045 with bad export cookie 8210181994358001405 [ 4371.772641] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4371.776499] LustreError: 134922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4372.171481] Lustre: server umount lustre-MDT0001 complete [ 4381.798557] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 13:10:53 (1788801053) [ 4387.653640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4403.168789] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4408.289462] LustreError: 137355:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 4434.016749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.502255] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4451.924564] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4454.254696] Lustre: server umount lustre-OST0000 complete [ 4464.408327] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 13:12:16 (1788801136) [ 4470.996049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.646899] LustreError: 138801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.740311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.565699] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4482.023889] LustreError: 138801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.041818] LustreError: 138801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4484.809798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4489.169346] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4496.299152] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4502.001436] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4508.956529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.268486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.273698] Lustre: Skipped 1 previous similar message [ 4515.210943] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4522.312686] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4531.169222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4531.181393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.206232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4531.215547] Lustre: Skipped 2 previous similar messages [ 4535.124353] Lustre: server umount lustre-MDT0000 complete [ 4537.319136] LustreError: 138798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.351731] LustreError: 138798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4538.824805] LustreError: 138783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801212 with bad export cookie 8210181994358002070 [ 4538.841954] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.852855] LustreError: 138783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4539.190819] Lustre: server umount lustre-MDT0001 complete [ 4552.892360] Lustre: server umount lustre-OST0000 complete [ 4560.557854] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 13:13:53 (1788801233) [ 4566.850373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.404791] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4572.644058] LustreError: 142455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4572.659956] LustreError: 142455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4580.309519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.800196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4580.811520] Lustre: Skipped 1 previous similar message [ 4585.842770] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4593.177801] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4598.307834] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4604.785045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.196117] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4612.343988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4617.326142] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4621.572586] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4621.755256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4626.257805] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4626.482173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4632.043308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.049346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.059760] Lustre: Skipped 2 previous similar messages [ 4632.067450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.074941] Lustre: Skipped 3 previous similar messages [ 4634.757989] Lustre: server umount lustre-OST0000 complete [ 4642.287675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4642.300933] LustreError: Skipped 1 previous similar message [ 4644.850767] Lustre: server umount lustre-MDT0000 complete [ 4648.389482] LustreError: 142436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801322 with bad export cookie 8210181994358003141 [ 4648.402645] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.411379] LustreError: 142436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4648.729097] Lustre: server umount lustre-MDT0001 complete [ 4656.782509] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 13:15:29 (1788801329) [ 4662.787750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4678.438767] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4683.552813] LustreError: 146612:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 4709.344430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.351937] Lustre: Skipped 1 previous similar message [ 4714.361169] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4721.019332] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4728.040776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.382323] LustreError: 146637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.397451] LustreError: 146637:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4728.581381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4732.846708] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4740.291853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.653801] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4754.112181] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4759.727541] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4764.693870] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4764.929974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4769.471459] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4769.770621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4774.886301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.887392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4774.926715] Lustre: Skipped 4 previous similar messages [ 4774.945955] Lustre: Skipped 7 previous similar messages [ 4778.127561] Lustre: server umount lustre-MDT0000 complete [ 4783.004934] LustreError: 146620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801456 with bad export cookie 8210181994358004163 [ 4783.014970] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4783.021658] LustreError: 146620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4783.540916] Lustre: server umount lustre-MDT0001 complete [ 4798.945441] Lustre: server umount lustre-OST0000 complete [ 4807.073746] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 13:17:59 (1788801479) [ 4813.867576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4829.408617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4834.529776] LustreError: 150805:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 4866.458266] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4874.874486] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4882.262776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.836217] LustreError: 150830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.865368] LustreError: 150830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4883.072284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4887.847461] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4896.840717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4902.888548] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4910.801954] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4916.426954] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4923.946261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4924.217687] Lustre: lustre-OST0001: new disk, initializing [ 4924.223231] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4924.229593] Lustre: 153680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4930.989581] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 4931.709646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4931.731601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4931.882694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4940.781629] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4945.236595] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4945.418447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4950.002508] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4950.199751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4952.552405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.562995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4952.581168] Lustre: Skipped 2 previous similar messages [ 4952.599047] Lustre: Skipped 1 previous similar message [ 4966.370218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4966.443790] Lustre: server umount lustre-OST0000 complete [ 4973.027449] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.452145] Lustre: server umount lustre-OST0001 complete [ 4986.693845] Lustre: server umount lustre-MDT0000 complete [ 4990.375295] LustreError: 150813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801664 with bad export cookie 8210181994358005115 [ 4990.400212] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4990.410200] LustreError: 150813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4990.769910] Lustre: server umount lustre-MDT0001 complete [ 5017.734879] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 13:21:29 (1788801689) [ 5019.744688] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5022.037537] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5024.417632] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 13:21:36 (1788801696) [ 5035.023751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5035.535441] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5035.625390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5036.017652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.026868] Lustre: Skipped 6 previous similar messages [ 5044.038884] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5054.500837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.631443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5054.698468] Lustre: 157241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5061.077104] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5070.254476] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5076.593565] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5083.746911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5084.020831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5090.507778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5091.639523] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5099.769982] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5104.871694] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5105.163209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5109.854336] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5110.167137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5115.877941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5115.887872] LustreError: Skipped 1 previous similar message [ 5115.890058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.909310] Lustre: Skipped 4 previous similar messages [ 5118.272581] Lustre: server umount lustre-OST0000 complete [ 5139.923069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5141.297316] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.342078] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5147.314467] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5154.856858] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5156.726694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.744213] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5160.488908] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5160.783971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5165.727582] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5165.980370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5171.904181] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c3650796000.ost_server_uuid 50 [ 5173.593346] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c3650796000.ost_server_uuid in FULL state after 0 sec [ 5173.783808] LustreError: 157246:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.8@tcp) returned error from blocking AST (req@0000000047b5be5c x1875693613095040 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000009a579d4/0x71f06d06430466d5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.8@tcp remote: 0xb1be915c96466969 expref: 5 pid: 157248 timeout: 5273 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5173.838308] LustreError: lustre-MDT0000: A client on nid 192.168.201.8@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5173.847716] LustreError: 157236:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.8@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000009a579d4/0x71f06d06430466d5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.8@tcp remote: 0xb1be915c96466969 expref: 6 pid: 157248 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5179.883951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5179.899980] LustreError: Skipped 1 previous similar message [ 5184.347384] Lustre: server umount lustre-OST0000 complete [ 5194.615962] Lustre: server umount lustre-MDT0000 complete [ 5197.799862] LustreError: 157251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.817370] LustreError: 157251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 5198.066020] LustreError: 157233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801871 with bad export cookie 8210181994358006298 [ 5198.068743] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.077300] LustreError: 157233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5204.496447] Lustre: server umount lustre-MDT0001 complete [ 5215.803841] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5218.829516] Key type lgssc unregistered [ 5219.161754] LNet: 163356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5219.171757] LNetError: 163356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5219.186452] LNet: Removed LNI 192.168.201.108@tcp [ 5220.145584] Key type .llcrypt unregistered [ 5220.149053] Key type ._llcrypt unregistered [ 5241.854272] Key type ._llcrypt registered [ 5241.858752] Key type .llcrypt registered [ 5241.951132] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5242.657731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5242.770086] alg: No test for adler32 (adler32-zlib) [ 5243.912308] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 5244.183225] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5245.848205] Key type lgssc registered [ 5246.993314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5253.642933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5253.673317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.075921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.257474] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5264.360839] LustreError: 164761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5264.383883] LustreError: 164761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5267.289702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.731511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5272.156908] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5280.731043] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5289.312324] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5296.173872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5296.675521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5299.787187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5304.650936] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5313.787875] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5321.766898] Lustre: Failing over lustre-MDT0000 [ 5322.195103] LustreError: 164755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.8@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5322.303574] Lustre: server umount lustre-MDT0000 complete [ 5323.240287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.249114] LustreError: 164761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.257741] LustreError: 164761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5328.354037] LustreError: 164760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5328.377976] LustreError: 164760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5333.475846] LustreError: 164760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.502778] LustreError: 164760:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5334.546209] LustreError: 164742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.8@tcp arrived at 1788802008 with bad export cookie 5408018230672200993 [ 5339.616135] Lustre: 164071:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788801997/real 1788801997] req@0000000096f8e379 x1875694759997056/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788802013 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5339.667166] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5341.808022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.740919] LustreError: 164756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5344.768610] LustreError: 164756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5350.193124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.278946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5355.016763] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5355.503203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5355.507753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5357.169264] Lustre: lustre-MDT0000: Denying connection for new client 23440163-2d82-4bf9-8bce-2ebb2378201f (at 192.168.201.8@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5365.734444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.739721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5365.745339] Lustre: Skipped 3 previous similar messages [ 5365.759855] Lustre: Skipped 1 previous similar message [ 5370.857852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5371.737742] Lustre: server umount lustre-OST0000 complete [ 5376.001736] LustreError: 168490:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5376.014918] Lustre: 167837:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5376.033802] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5376.061320] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5376.075304] LustreError: 167837:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002be176ac x1875694760017920/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5376.100132] LustreError: 167837:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5376.109170] LustreError: 167837:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5376.121861] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5376.814532] Lustre: server umount lustre-MDT0000 complete [ 5381.097804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.138047] LustreError: 164756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.163300] LustreError: 164756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5381.217609] LustreError: 164742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802055 with bad export cookie 5408018230672201091 [ 5381.225300] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5381.233623] LustreError: 164742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5387.722213] Lustre: server umount lustre-MDT0001 complete [ 5399.586590] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5402.334545] Key type lgssc unregistered [ 5402.706549] LNet: 169245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5402.715340] LNetError: 169245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5402.730852] LNet: Removed LNI 192.168.201.108@tcp [ 5403.829515] Key type .llcrypt unregistered [ 5403.834586] Key type ._llcrypt unregistered [ 5431.322488] Key type ._llcrypt registered [ 5431.332677] Key type .llcrypt registered [ 5431.548510] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5433.622094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5433.718600] alg: No test for adler32 (adler32-zlib) [ 5435.110883] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 5435.474658] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5437.272479] Key type lgssc registered [ 5438.321665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5447.139556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5447.163647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.661323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.825450] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5458.924696] LustreError: 170654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.945954] LustreError: 170654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5461.772319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.294276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5467.045694] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5475.059790] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5480.906549] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.112697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5489.394787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5493.497434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5497.354428] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5505.154480] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5513.185401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5513.194853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.196406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5513.213266] Lustre: Skipped 1 previous similar message [ 5518.117254] Lustre: server umount lustre-OST0000 complete [ 5523.426788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.446055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5523.450328] Lustre: Skipped 2 previous similar messages [ 5527.857890] Lustre: server umount lustre-MDT0000 complete [ 5528.547645] LustreError: 171664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.569175] LustreError: 171664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5532.061016] LustreError: 170636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802205 with bad export cookie 12490890697020395647 [ 5532.070525] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.080958] LustreError: 170636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5532.799703] Lustre: server umount lustre-MDT0001 complete [ 5545.829726] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5549.322088] Key type lgssc unregistered [ 5549.750200] LNet: 174189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5549.763505] LNetError: 174189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5549.783208] LNet: Removed LNI 192.168.201.108@tcp [ 5551.007844] Key type .llcrypt unregistered [ 5551.012789] Key type ._llcrypt unregistered [ 5585.882793] Key type ._llcrypt registered [ 5585.890164] Key type .llcrypt registered [ 5586.051540] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5587.367069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5587.441267] alg: No test for adler32 (adler32-zlib) [ 5588.595698] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 5588.894250] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5590.600690] Key type lgssc registered [ 5592.043419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5600.899493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5600.913632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.377866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.014995] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5612.516167] LustreError: 175889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.557057] LustreError: 175889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5616.689202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5617.096278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5622.293587] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5629.315919] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5636.070491] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.190236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.589896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.659191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5652.326241] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5660.586863] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.417880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.421556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5672.448359] Lustre: Skipped 1 previous similar message [ 5672.467301] Lustre: Skipped 1 previous similar message [ 5674.680809] Lustre: server umount lustre-OST0000 complete [ 5682.661150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.683151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.923655] Lustre: server umount lustre-MDT0000 complete [ 5687.781727] LustreError: 175884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.797963] LustreError: 175884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5688.982361] LustreError: 175870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802362 with bad export cookie 11605441934265901372 [ 5689.003171] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.510279] Lustre: server umount lustre-MDT0001 complete [ 5701.284975] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5703.848334] Key type lgssc unregistered [ 5704.146950] LNet: 179422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.152935] LNetError: 179422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.166503] LNet: Removed LNI 192.168.201.108@tcp [ 5705.233343] Key type .llcrypt unregistered [ 5705.234954] Key type ._llcrypt unregistered [ 5727.155801] Key type ._llcrypt registered [ 5727.159049] Key type .llcrypt registered [ 5727.257903] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5728.598159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5728.747495] alg: No test for adler32 (adler32-zlib) [ 5729.915615] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 5730.267661] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5732.008735] Key type lgssc registered [ 5733.455575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5746.525262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5746.570811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5748.055787] Lustre: *** cfs_fail_loc=135, val=0*** [ 5748.152857] LustreError: 180942:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5748.161268] LustreError: 180942:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.108@tcp: cfg command failed: rc = -2 [ 5748.168293] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5748.168293] [ 5748.175159] LustreError: MGC192.168.201.108@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5748.184237] LustreError: 180904:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5748.190254] LustreError: 180904:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5748.197257] LustreError: 180904:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5748.204369] LustreError: 180904:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5748.281751] Lustre: server umount lustre-MDT0000 complete [ 5748.286961] LustreError: 180904:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5755.470401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.968186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.111686] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5765.098472] LustreError: 181313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.134614] LustreError: 181313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5770.212817] LustreError: 181312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.353265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.859316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5776.676768] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5784.523266] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5790.609796] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5796.903814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5796.908911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5796.915899] Lustre: Skipped 1 previous similar message [ 5796.946355] Lustre: Skipped 1 previous similar message [ 5799.207939] Lustre: server umount lustre-MDT0000 complete [ 5801.958338] LustreError: 181309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.988729] LustreError: 181309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5803.755093] LustreError: 181293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802477 with bad export cookie 8843141894940151802 [ 5803.775189] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5804.125248] Lustre: server umount lustre-MDT0001 complete [ 5815.044328] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5818.130929] Key type lgssc unregistered [ 5818.449888] LNet: 183635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5818.467628] LNetError: 183635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5819.496937] LNet: Removed LNI 192.168.201.108@tcp [ 5820.415238] Key type .llcrypt unregistered [ 5820.420209] Key type ._llcrypt unregistered [ 5841.705610] Key type ._llcrypt registered [ 5841.707322] Key type .llcrypt registered [ 5841.887204] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 5842.349148] Key type .llcrypt unregistered [ 5842.350860] Key type ._llcrypt unregistered [ 5864.393481] Key type ._llcrypt registered [ 5864.395494] Key type .llcrypt registered [ 5864.547928] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 5865.450983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5865.485408] alg: No test for adler32 (adler32-zlib) [ 5866.747655] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 5867.175420] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 5868.888217] Key type lgssc registered [ 5870.288601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5879.425412] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5879.439226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5894.944615] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5900.064419] LustreError: 185764:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.108@tcp: failed processing log, type 4: rc = -110 [ 5926.752324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.968865] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5944.065974] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5951.797347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.533429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.617062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5957.543761] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5957.613377] LustreError: 185789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.637589] LustreError: 185789:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5961.699586] LustreError: 186801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.718892] LustreError: 186801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5966.823314] LustreError: 186800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.856538] LustreError: 186800:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5966.903816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.436561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.745322] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 5981.604261] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5987.836329] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5991.148326] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6003.181938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.191030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.205503] Lustre: Skipped 1 previous similar message [ 6003.240022] Lustre: Skipped 1 previous similar message [ 6006.514033] Lustre: server umount lustre-OST0000 complete [ 6013.411101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.436178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6013.443401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6016.952385] Lustre: server umount lustre-MDT0000 complete [ 6018.531553] LustreError: 186801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6018.552285] LustreError: 186801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6021.811026] LustreError: 185772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802695 with bad export cookie 444254204024463709 [ 6021.813098] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6021.837544] LustreError: 185772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6022.248324] Lustre: server umount lustre-MDT0001 complete [ 6033.042938] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6036.176124] Key type lgssc unregistered [ 6036.453541] LNet: 189681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6036.458242] LNetError: 189681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6036.477919] LNet: Removed LNI 192.168.201.108@tcp [ 6037.485295] Key type .llcrypt unregistered [ 6037.490866] Key type ._llcrypt unregistered [ 6063.632769] Key type ._llcrypt registered [ 6063.634795] Key type .llcrypt registered [ 6063.715564] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6065.140821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6065.200840] alg: No test for adler32 (adler32-zlib) [ 6066.240442] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6066.406452] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6068.056232] Key type lgssc registered [ 6069.392387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6078.053498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6078.089908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.790718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6085.428102] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6089.188927] LustreError: 191093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.200467] LustreError: 191093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6094.315463] LustreError: 191092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.318458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.899996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6100.474882] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6108.726181] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6117.021538] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6125.091324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6125.431384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.675751] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6139.775395] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6150.016766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6156.464115] Lustre: Failing over lustre-MDT0000 [ 6156.768637] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.8@tcp (stopping) [ 6156.961916] Lustre: server umount lustre-MDT0000 complete [ 6160.354040] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.356582] LustreError: 191088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6160.370440] Lustre: Skipped 1 previous similar message [ 6160.403510] LustreError: 191088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6165.476958] LustreError: 191089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6165.492579] LustreError: 191089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6170.598916] LustreError: 192028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.629947] LustreError: 192028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6176.800453] Lustre: 190401:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788802834/real 1788802834] req@00000000e771c516 x1875695621927424/t0(0) o400->MGC192.168.201.108@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788802850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6176.830747] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.800383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.860099] LustreError: 191086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.914069] LustreError: 191086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6186.982574] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1bb65081ef7441a9 [ 6187.017522] Lustre: MGC192.168.201.108@tcp: Connection restored to 0@lo (at 0@lo) [ 6187.437777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6187.541627] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6192.597496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6192.631558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6192.653169] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6192.700243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6192.876671] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6197.986503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6206.278061] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6216.673612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6216.688371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6216.715650] Lustre: Skipped 1 previous similar message [ 6216.727187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.144439] Lustre: server umount lustre-OST0000 complete [ 6223.337927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.345113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6223.353386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.361407] Lustre: Skipped 1 previous similar message [ 6228.449397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.453029] Lustre: Skipped 1 previous similar message [ 6230.621947] Lustre: server umount lustre-MDT0000 complete [ 6233.574661] LustreError: 191092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.591582] LustreError: 191092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6235.891904] LustreError: 191074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802909 with bad export cookie 1996872003783770537 [ 6235.895553] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6235.901492] LustreError: 191074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6236.401654] Lustre: server umount lustre-MDT0001 complete [ 6247.791728] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6250.927684] Key type lgssc unregistered [ 6251.192609] LNet: 196005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6251.202140] LNetError: 196005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6251.220593] LNet: Removed LNI 192.168.201.108@tcp [ 6252.132473] Key type .llcrypt unregistered [ 6252.134997] Key type ._llcrypt unregistered [ 6278.635378] Key type ._llcrypt registered [ 6278.638724] Key type .llcrypt registered [ 6278.774643] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6280.050500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6280.160829] alg: No test for adler32 (adler32-zlib) [ 6281.429346] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6281.816546] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6283.520223] Key type lgssc registered [ 6285.646995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6297.125844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6297.139926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6298.875538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6302.964626] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6308.326157] LustreError: 197414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6308.338742] LustreError: 197414:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6312.706806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.116974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6318.126157] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6326.564582] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6333.528047] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6340.969467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.387943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.475945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6347.761599] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6355.738329] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.069971] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6363.942275] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6381.668470] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6389.731660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6389.747107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6389.769198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.827260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6393.834516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6393.837487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6393.863968] Lustre: Skipped 1 previous similar message [ 6395.795278] Lustre: server umount lustre-OST0000 complete [ 6404.069168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.082481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6406.626973] Lustre: server umount lustre-MDT0000 complete [ 6409.188094] LustreError: 197414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.218965] LustreError: 197414:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6411.365304] LustreError: 197395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803085 with bad export cookie 1052858624345906960 [ 6411.372234] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.382243] LustreError: 197395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6411.910326] Lustre: server umount lustre-MDT0001 complete [ 6424.291152] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6427.344332] Key type lgssc unregistered [ 6427.673573] LNet: 201279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6427.686962] LNetError: 201279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6428.713325] LNet: Removed LNI 192.168.201.108@tcp [ 6429.416622] Key type .llcrypt unregistered [ 6429.418669] Key type ._llcrypt unregistered [ 6448.551677] Key type ._llcrypt registered [ 6448.554114] Key type .llcrypt registered [ 6448.676309] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6450.147811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6450.179392] alg: No test for adler32 (adler32-zlib) [ 6451.197719] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6451.432170] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6453.104235] Key type lgssc registered [ 6454.132472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6461.642904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6461.664909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.102576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.580550] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6472.674504] LustreError: 202544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6472.707533] LustreError: 202544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6475.754175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.273875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6480.735395] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6488.332913] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6493.443260] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6500.240363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.608817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.067814] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6507.773642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6515.304198] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6522.427781] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6536.472924] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6547.677488] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6555.234752] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6563.299992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6563.317327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.340772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6564.329383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6564.331072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.345081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6568.794668] Lustre: server umount lustre-OST0000 complete [ 6573.538101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6573.543276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.558797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6573.562933] Lustre: Skipped 1 previous similar message [ 6578.845461] Lustre: server umount lustre-MDT0000 complete [ 6579.683822] LustreError: 202544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.712719] LustreError: 202544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6582.608655] LustreError: 202525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803256 with bad export cookie 12571072834774616441 [ 6582.619537] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6582.621713] LustreError: 202525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6583.130962] Lustre: server umount lustre-MDT0001 complete [ 6593.867169] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6597.132443] Key type lgssc unregistered [ 6597.377450] LNet: 207516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6597.381085] LNetError: 207516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6597.406789] LNet: Removed LNI 192.168.201.108@tcp [ 6598.282658] Key type .llcrypt unregistered [ 6598.285036] Key type ._llcrypt unregistered [ 6623.686610] Key type ._llcrypt registered [ 6623.688657] Key type .llcrypt registered [ 6623.825364] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6624.932850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6624.969977] alg: No test for adler32 (adler32-zlib) [ 6626.194210] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6626.469573] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6628.209175] Key type lgssc registered [ 6629.489229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6636.950162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6636.970334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.491738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.038229] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6648.806807] LustreError: 208926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6648.820842] LustreError: 208926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6649.830674] LustreError: 208925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.660521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.123054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6656.881710] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6664.599674] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6670.921329] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6678.304371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6678.904046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.049900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6686.075912] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6693.878223] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6710.760148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6710.767317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6710.789463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6711.784688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6711.794852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6711.804156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6715.024152] Lustre: server umount lustre-OST0000 complete [ 6722.017405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.020330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6722.056541] Lustre: Skipped 1 previous similar message [ 6722.070529] Lustre: Skipped 2 previous similar messages [ 6724.905695] Lustre: server umount lustre-MDT0000 complete [ 6727.146507] LustreError: 208926:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.160498] LustreError: 208926:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6729.219313] LustreError: 208905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803403 with bad export cookie 13236865667290494918 [ 6729.232287] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6729.545650] Lustre: server umount lustre-MDT0001 complete [ 6739.660855] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6742.845521] Key type lgssc unregistered [ 6743.110413] LNet: 212604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6743.125437] LNetError: 212604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6743.140068] LNet: Removed LNI 192.168.201.108@tcp [ 6744.057803] Key type .llcrypt unregistered [ 6744.062099] Key type ._llcrypt unregistered [ 6763.286759] Key type ._llcrypt registered [ 6763.290669] Key type .llcrypt registered [ 6763.395536] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6764.369255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6764.524100] alg: No test for adler32 (adler32-zlib) [ 6765.600360] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6765.782910] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6767.456211] Key type lgssc registered [ 6768.812855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6777.329508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6777.375794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.026982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.825523] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6788.586287] LustreError: 213870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6788.608408] LustreError: 213870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6793.152893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.468042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6799.508245] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6809.523358] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6815.659920] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6823.476713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.764402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.390757] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6837.205463] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6848.398234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6863.844341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6863.853539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.859365] LustreError: Skipped 1 previous similar message [ 6863.862259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6863.870356] Lustre: Skipped 1 previous similar message [ 6865.911145] Lustre: server umount lustre-OST0000 complete [ 6874.082369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.095912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6874.112194] Lustre: Skipped 1 previous similar message [ 6874.123861] Lustre: Skipped 2 previous similar messages [ 6879.202440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6879.218730] Lustre: Skipped 1 previous similar message [ 6883.301291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6883.315833] Lustre: Skipped 1 previous similar message [ 6885.344205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6885.673885] Lustre: server umount lustre-MDT0000 complete [ 6888.424649] LustreError: 213864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.461393] LustreError: 213864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6889.913789] LustreError: 213850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803563 with bad export cookie 12669213983411018245 [ 6889.925308] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6889.930698] LustreError: 213850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6890.283905] Lustre: server umount lustre-MDT0001 complete [ 6901.188292] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 6903.998281] Key type lgssc unregistered [ 6904.411624] LNet: 217576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6904.422911] LNetError: 217576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6904.452369] LNet: Removed LNI 192.168.201.108@tcp [ 6905.385931] Key type .llcrypt unregistered [ 6905.391263] Key type ._llcrypt unregistered [ 6925.912614] Key type ._llcrypt registered [ 6925.921172] Key type .llcrypt registered [ 6926.123523] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 6927.874239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6927.942458] alg: No test for adler32 (adler32-zlib) [ 6929.316163] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 6929.643804] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 6931.312504] Key type lgssc registered [ 6932.476143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6940.222805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6940.238488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.767628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.783498] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6951.907315] LustreError: 218839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6951.927196] LustreError: 218839:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6954.885855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.323273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6961.415870] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 6970.805792] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6978.301914] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6986.362289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6986.794494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6992.753726] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7000.654744] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7011.434521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7022.835035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7022.949156] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7023.201644] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7030.696242] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7040.678470] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7042.933597] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7054.773847] Lustre: Permanently deactivating lustre-OST0001 [ 7054.799502] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7060.549557] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7067.525879] Lustre: Permanently reactivating lustre-OST0001 [ 7067.551236] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7067.556609] Lustre: Skipped 2 previous similar messages [ 7077.518932] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.543903] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7077.567625] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7077.589562] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7077.640156] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7079.226625] Lustre: server umount lustre-OST0001 complete [ 7080.939082] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.948305] Lustre: Skipped 2 previous similar messages [ 7080.949531] LustreError: 220663:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.961437] LustreError: 220663:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7083.489699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7083.497297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7083.507062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.049839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7086.055612] LustreError: 220661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7086.056355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.085695] LustreError: 220661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7089.919618] Lustre: server umount lustre-OST0000 complete [ 7096.289676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.307306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.316590] Lustre: Skipped 2 previous similar messages [ 7100.384425] Lustre: server umount lustre-MDT0000 complete [ 7101.430606] LustreError: 218836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7101.449047] LustreError: 218836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7104.046126] LustreError: 218821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803777 with bad export cookie 17085118876055207169 [ 7104.063365] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.070237] LustreError: 218821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7104.448659] Lustre: server umount lustre-MDT0001 complete [ 7115.033676] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7117.617034] Key type lgssc unregistered [ 7117.946410] LNet: 223905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7117.960624] LNetError: 223905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7117.984845] LNet: Removed LNI 192.168.201.108@tcp [ 7118.899426] Key type .llcrypt unregistered [ 7118.903255] Key type ._llcrypt unregistered [ 7161.142330] Key type ._llcrypt registered [ 7161.149936] Key type .llcrypt registered [ 7161.360380] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7163.031053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7163.077514] alg: No test for adler32 (adler32-zlib) [ 7164.241644] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7164.516586] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7166.240443] Key type lgssc registered [ 7167.428272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7175.493865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7175.506797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.787113] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7176.827507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7177.007798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7181.823213] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7189.784634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.852166] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7189.872851] Lustre: 226105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7190.054658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7194.950408] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7203.228640] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7208.826333] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.769727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.014190] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7216.268192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7221.906815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7224.335683] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7232.106807] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7236.338907] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7242.988388] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7253.440335] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7261.238867] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7278.245554] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7278.260373] Lustre: Skipped 1 previous similar message [ 7296.065848] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7296.071209] Lustre: Skipped 1 previous similar message [ 7332.429174] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7332.438687] Lustre: Skipped 4 previous similar messages [ 7402.645275] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7402.652459] Lustre: Skipped 8 previous similar messages [ 7411.172594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7411.178373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.201826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7411.682846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.713738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7415.699849] Lustre: server umount lustre-OST0000 complete [ 7420.385292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7420.388944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.414393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.837931] Lustre: server umount lustre-MDT0000 complete [ 7430.480673] LustreError: 228678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804104 with bad export cookie 7929793197783883714 [ 7430.492460] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7430.493285] LustreError: 228678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7431.251875] Lustre: server umount lustre-MDT0001 complete [ 7444.859720] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7448.074314] Key type lgssc unregistered [ 7448.316785] LNet: 230898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7448.323419] LNetError: 230898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7448.336418] LNet: Removed LNI 192.168.201.108@tcp [ 7449.529793] Key type .llcrypt unregistered [ 7449.534192] Key type ._llcrypt unregistered [ 7475.556113] Key type ._llcrypt registered [ 7475.562994] Key type .llcrypt registered [ 7475.753750] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7476.887872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7476.982098] alg: No test for adler32 (adler32-zlib) [ 7478.341163] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7478.839610] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7480.560229] Key type lgssc registered [ 7481.813747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7490.561959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7490.579153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.127773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.723493] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7502.354027] LustreError: 232310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.384500] LustreError: 232310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7506.343265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7506.864304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7512.723318] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7521.922412] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7528.505518] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7538.259227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7538.795965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7547.107765] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_default_debug -1 all [ 7557.067304] Lustre: DEBUG MARKER: oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7562.361419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7583.201222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7583.210627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.228814] LustreError: Skipped 1 previous similar message [ 7583.232431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7583.294464] Lustre: Skipped 1 previous similar message [ 7587.856549] Lustre: server umount lustre-OST0000 complete [ 7593.442455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.446704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.475884] Lustre: Skipped 1 previous similar message [ 7593.495675] Lustre: Skipped 2 previous similar messages [ 7598.561547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.570363] Lustre: Skipped 1 previous similar message [ 7603.682103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7603.691109] Lustre: Skipped 1 previous similar message [ 7606.240149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7606.358063] Lustre: server umount lustre-MDT0000 complete [ 7608.801676] LustreError: 233038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.828397] LustreError: 233038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7610.492918] LustreError: 232292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804284 with bad export cookie 16360381440143913594 [ 7610.503991] LustreError: MGC192.168.201.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.509403] LustreError: 232292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7610.973919] Lustre: server umount lustre-MDT0001 complete [ 7624.301805] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7627.498715] Key type lgssc unregistered [ 7627.753621] LNet: 236007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7627.758674] LNetError: 236007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7627.791478] LNet: Removed LNI 192.168.201.108@tcp [ 7628.861108] Key type .llcrypt unregistered [ 7628.865201] Key type ._llcrypt unregistered [ 7653.977966] Key type ._llcrypt registered [ 7653.979591] Key type .llcrypt registered [ 7654.131499] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7655.335573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.528526] alg: No test for adler32 (adler32-zlib) [ 7656.703677] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7656.951601] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7658.657259] Key type lgssc registered [ 7660.064870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7930.873719] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 7933.969541] Key type lgssc unregistered [ 7934.403308] LNet: 238032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7934.409849] LNetError: 238032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7934.428298] LNet: Removed LNI 192.168.201.108@tcp [ 7935.417588] Key type .llcrypt unregistered [ 7935.420289] Key type ._llcrypt unregistered [ 7963.956820] Key type ._llcrypt registered [ 7963.959364] Key type .llcrypt registered [ 7964.097850] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing set_hostid [ 7984.768509] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing load_modules_local [ 7986.255628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7986.582824] alg: No test for adler32 (adler32-zlib) [ 7987.656556] Lustre: Lustre: Build Version: 2.17.58_39_gc924a9e [ 7987.952226] LNet: Added LNI 192.168.201.108@tcp [8/256/0/180] [ 7989.716206] Key type lgssc registered [ 7991.022706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8027.940165] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7714 sec ========== 14:11:39 (1788804699) [ 8030.454422] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:11:41 (1788804701) === [ 8035.004402] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:11:46 (1788804706) === [ 8087.850965] Lustre: DEBUG MARKER: oleg108-server.virtnet: executing unload_modules_local [ 8091.416288] Key type lgssc unregistered [ 8091.798941] LNet: 243271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8091.805069] LNetError: 243271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8091.817801] LNet: Removed LNI 192.168.201.108@tcp [ 8092.976556] Key type .llcrypt unregistered [ 8092.978327] Key type ._llcrypt unregistered