[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520653062 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003176] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008333] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012122] LSM: Security Framework initializing [ 0.013076] Yama: becoming mindful. [ 0.014042] SELinux: Initializing. [ 0.015100] *** VALIDATE selinux *** [ 0.024566] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029280] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032027] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033156] *** VALIDATE tmpfs *** [ 0.035195] *** VALIDATE proc *** [ 0.036264] *** VALIDATE cgroup *** [ 0.038010] *** VALIDATE cgroup2 *** [ 0.039290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041033] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044018] Spectre V2 : User space: Vulnerable [ 0.045012] Speculative Store Bypass: Vulnerable [ 0.047804] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.049136] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050603] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051029] ... version: 2 [ 0.052014] ... bit width: 48 [ 0.053015] ... generic registers: 4 [ 0.054015] ... value mask: 0000ffffffffffff [ 0.055018] ... max period: 00007fffffffffff [ 0.056017] ... fixed-purpose events: 3 [ 0.057012] ... event mask: 000000070000000f [ 0.058363] rcu: Hierarchical SRCU implementation. [ 0.060221] smp: Bringing up secondary CPUs ... [ 0.061527] x86: Booting SMP configuration: [ 0.062025] .... node #0, CPUs: #1 #2 #3 [ 0.066282] smp: Brought up 1 node, 4 CPUs [ 0.068016] smpboot: Max logical packages: 1 [ 0.069015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.183092] node 0 deferred pages initialised in 111ms [ 0.186180] devtmpfs: initialized [ 0.188362] x86/mm: Memory block size: 128MB [ 0.192308] gcov: version magic: 0x41383552 [ 0.195409] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.199111] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.201460] pinctrl core: initialized pinctrl subsystem [ 0.203319] [ 0.203868] ************************************************************* [ 0.206019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208015] ** ** [ 0.210024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.212014] ** ** [ 0.214057] ** This means that this kernel is built to expose internal ** [ 0.216021] ** IOMMU data structures, which may compromise security on ** [ 0.218021] ** your system. ** [ 0.221025] ** ** [ 0.223020] ** If you see this message and you are not debugging the ** [ 0.226021] ** kernel, report this immediately to your vendor! ** [ 0.228011] ** ** [ 0.229010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.232019] ************************************************************* [ 0.235038] NET: Registered protocol family 16 [ 0.237596] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.240103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.243082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.247656] cpuidle: using governor menu [ 0.250181] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.252867] PCI: Using configuration type 1 for base access [ 0.255168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.263136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.265062] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.269127] cryptd: max_cpu_qlen set to 1000 [ 0.271272] ACPI: Added _OSI(Module Device) [ 0.273022] ACPI: Added _OSI(Processor Device) [ 0.274021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.276017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280566] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286409] ACPI: Interpreter enabled [ 0.288062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.289016] ACPI: Using IOAPIC for interrupt routing [ 0.291156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.293483] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.303885] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.305062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.307022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.310118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.314795] acpiphp: Slot [2] registered [ 0.315237] acpiphp: Slot [5] registered [ 0.317267] acpiphp: Slot [6] registered [ 0.318123] acpiphp: Slot [7] registered [ 0.319000] acpiphp: Slot [8] registered [ 0.321196] acpiphp: Slot [9] registered [ 0.322183] acpiphp: Slot [10] registered [ 0.323185] acpiphp: Slot [3] registered [ 0.325137] acpiphp: Slot [4] registered [ 0.327171] acpiphp: Slot [11] registered [ 0.328260] acpiphp: Slot [12] registered [ 0.330299] acpiphp: Slot [13] registered [ 0.332342] acpiphp: Slot [14] registered [ 0.334147] acpiphp: Slot [15] registered [ 0.336153] acpiphp: Slot [16] registered [ 0.337129] acpiphp: Slot [17] registered [ 0.339159] acpiphp: Slot [18] registered [ 0.340230] acpiphp: Slot [19] registered [ 0.342128] acpiphp: Slot [20] registered [ 0.343140] acpiphp: Slot [21] registered [ 0.345171] acpiphp: Slot [22] registered [ 0.346142] acpiphp: Slot [23] registered [ 0.348150] acpiphp: Slot [24] registered [ 0.349139] acpiphp: Slot [25] registered [ 0.350149] acpiphp: Slot [26] registered [ 0.352191] acpiphp: Slot [27] registered [ 0.354148] acpiphp: Slot [28] registered [ 0.355148] acpiphp: Slot [29] registered [ 0.356148] acpiphp: Slot [30] registered [ 0.358151] acpiphp: Slot [31] registered [ 0.359097] PCI host bridge to bus 0000:00 [ 0.361034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364116] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.366042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.369034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.370039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.372037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.374250] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.376436] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.379755] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.389817] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.396024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.398027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.400501] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403816] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.405055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.408978] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.416020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428035] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.434020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441010] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.469032] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.477684] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.492046] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.523032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.540050] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552069] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560074] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.582027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596220] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.607026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680257] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.693031] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.706045] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.739022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.761728] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.774036] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.789038] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.829037] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.849749] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.851393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.853351] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.855311] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.857230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.862048] iommu: Default domain type: Passthrough [ 0.864623] SCSI subsystem initialized [ 0.866123] ACPI: bus type USB registered [ 0.867119] usbcore: registered new interface driver usbfs [ 0.869083] usbcore: registered new interface driver hub [ 0.871097] usbcore: registered new device driver usb [ 0.872000] pps_core: LinuxPPS API ver. 1 registered [ 0.872000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.875062] PTP clock support registered [ 0.877163] EDAC MC: Ver: 3.0.0 [ 0.879131] PCI: Using ACPI for IRQ routing [ 0.880788] NetLabel: Initializing [ 0.881010] NetLabel: domain hash size = 128 [ 0.883010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.884083] NetLabel: unlabeled traffic allowed by default [ 0.886189] vgaarb: loaded [ 0.888285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.889019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.895006] clocksource: Switched to clocksource kvm-clock [ 1.018335] VFS: Disk quotas dquot_6.6.0 [ 1.020432] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.023304] *** VALIDATE ramfs *** [ 1.024656] *** VALIDATE hugetlbfs *** [ 1.026680] pnp: PnP ACPI init [ 1.029562] pnp: PnP ACPI: found 6 devices [ 1.046040] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.048648] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.050329] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.052103] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.053883] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.055781] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.058232] NET: Registered protocol family 2 [ 1.060534] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.065632] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.068815] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.073750] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.077245] TCP: Hash tables configured (established 65536 bind 65536) [ 1.079915] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.082728] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.085598] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.088750] NET: Registered protocol family 1 [ 1.093179] RPC: Registered named UNIX socket transport module. [ 1.095504] RPC: Registered udp transport module. [ 1.097184] RPC: Registered tcp transport module. [ 1.098747] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.101081] NET: Registered protocol family 44 [ 1.102807] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.104930] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.107108] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.109271] PCI: CLS 0 bytes, default 64 [ 1.111254] Unpacking initramfs... [ 2.687691] debug: unmapping init [mem 0xffff92533cc54000-0xffff92533ffbffff] [ 2.690627] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.692278] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.694592] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.210855] Initialise system trusted keyrings [ 3.212436] Key type blacklist registered [ 3.214125] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.223166] zbud: loaded [ 3.226245] *** VALIDATE nfs *** [ 3.227488] *** VALIDATE nfs4 *** [ 3.229186] pstore: using deflate compression [ 3.233440] Platform Keyring initialized [ 3.408660] NET: Registered protocol family 38 [ 3.410460] Key type asymmetric registered [ 3.412136] Asymmetric key parser 'x509' registered [ 3.413941] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.417195] io scheduler mq-deadline registered [ 3.418806] io scheduler kyber registered [ 3.420976] io scheduler bfq registered [ 3.423677] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.429225] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.432689] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.436354] ACPI: Power Button [PWRF] [ 3.442552] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.451222] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.468760] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.481802] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.517880] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.547606] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.575180] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.579353] Non-volatile memory driver v1.3 [ 3.580892] Linux agpgart interface v0.103 [ 3.613262] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.616187] vda: detected capacity change from 0 to 76775424 [ 3.649196] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.652488] vdb: detected capacity change from 0 to 1073741824 [ 3.675282] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.677771] vdc: detected capacity change from 0 to 2621440000 [ 3.698101] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.700627] vdd: detected capacity change from 0 to 2621440000 [ 3.713213] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.715258] vde: detected capacity change from 0 to 4294967296 [ 3.727311] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.729432] vdf: detected capacity change from 0 to 4294967296 [ 3.734315] libphy: Fixed MDIO Bus: probed [ 3.742489] usbcore: registered new interface driver usbserial_generic [ 3.745875] usbserial: USB Serial support registered for generic [ 3.748453] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.752373] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.754305] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.756780] mousedev: PS/2 mouse device common for all mice [ 3.760245] rtc_cmos 00:05: RTC can wake from S4 [ 3.762889] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.765882] rtc_cmos 00:05: registered as rtc0 [ 3.772525] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.773274] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.776414] intel_pstate: CPU model not supported [ 3.784738] hid: raw HID events driver (C) Jiri Kosina [ 3.786935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.787111] usbcore: registered new interface driver usbhid [ 3.792280] usbhid: USB HID core driver [ 3.794239] drop_monitor: Initializing network drop monitor service [ 3.797079] Initializing XFRM netlink socket [ 3.798767] NET: Registered protocol family 10 [ 3.802613] Segment Routing with IPv6 [ 3.804686] NET: Registered protocol family 17 [ 3.808192] mpls_gso: MPLS GSO support [ 3.814568] RAS: Correctable Errors collector initialized. [ 3.816765] AVX version of gcm_enc/dec engaged. [ 3.818406] AES CTR mode by8 optimization enabled [ 3.894460] sched_clock: Marking stable (3894427871, 0)->(4868151579, -973723708) [ 3.897355] registered taskstats version 1 [ 3.898900] Loading compiled-in X.509 certificates [ 3.900805] zswap: loaded using pool lzo/zbud [ 3.930373] Key type big_key registered [ 3.944701] Key type encrypted registered [ 3.946196] ima: No TPM chip found, activating TPM-bypass! [ 3.947876] ima: Allocated hash algorithm: sha1 [ 3.949890] ima: No architecture policies found [ 3.951626] evm: Initialising EVM extended attributes: [ 3.953189] evm: security.selinux [ 3.954363] evm: security.ima [ 3.955137] evm: security.capability [ 3.956198] evm: HMAC attrs: 0x1 [ 3.960274] rtc_cmos 00:05: setting system clock to 2026-09-10 07:09:19 UTC (1789024159) [ 3.966708] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 3.970105] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 3.978182] Write protecting the kernel read-only data: 28672k [ 3.981447] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 3.983642] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 4.027277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.036415] systemd[1]: Detected virtualization kvm. [ 4.038545] systemd[1]: Detected architecture x86-64. [ 4.040614] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.062168] systemd[1]: No hostname configured. [ 4.064146] systemd[1]: Set hostname to . [ 4.065847] random: systemd: uninitialized urandom read (16 bytes read) [ 4.068355] systemd[1]: Initializing machine ID from random generator. [ 4.220383] random: systemd: uninitialized urandom read (16 bytes read) [ 4.222894] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.227165] random: systemd: uninitialized urandom read (16 bytes read) [ 4.229678] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.233977] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.910374] device-mapper: uevent: version 1.0.3 [ 4.912603] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.821070] virtio_net virtio0 ens2: renamed from eth0 [ 5.879887] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.000324] scsi host0: ata_piix [ 6.032650] scsi host1: ata_piix [ 6.037747] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.044641] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.478857] random: crng init done [ 10.481523] random: 7 urandom warning(s) missed due to ratelimiting [ 12.765902] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.187407] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.745766] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.890847] SELinux: Disabled at runtime. [ 20.077427] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.102094] systemd[1]: Detected virtualization kvm. [ 20.103922] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.196590] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.215201] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.235637] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.238366] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.253676] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.290555] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.336967] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 23.420628] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.927491] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.276777] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.437836] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.164895] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.350514] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 34.692670] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 35.739584] NFS: Registering the id_resolver key type [ 35.745162] Key type id_resolver registered [ 35.748387] Key type id_legacy registered [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 40.761317] hrtimer: interrupt took 24128180 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg250-server login: [ 98.371198] libcfs: loading out-of-tree module taints kernel. [ 98.431231] Key type ._llcrypt registered [ 98.435676] Key type .llcrypt registered [ 98.575156] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 121.211991] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 122.693594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 122.715805] alg: No test for adler32 (adler32-zlib) [ 124.463373] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 125.566463] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 127.423184] Key type lgssc registered [ 128.969417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 149.409812] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.122663] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 207.123463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 207.173380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.718600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.770856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.890818] Lustre: lustre-MDT0000: new disk, initializing [ 208.970523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.994227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 214.041660] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.208904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 230.360745] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.408100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.414225] Lustre: Skipped 1 previous similar message [ 230.548603] Lustre: lustre-MDT0001: new disk, initializing [ 230.662696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.752488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.769546] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 236.515273] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.904275] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 254.172221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.543562] Lustre: lustre-OST0000: new disk, initializing [ 254.555259] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 254.567864] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 254.656065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.195215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 260.212274] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 260.348539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.217885] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.047257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.262267] Lustre: lustre-OST0001: new disk, initializing [ 280.275723] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.283328] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.410299] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.292143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 286.306608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 286.397810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 287.504907] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 299.593522] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.859615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 319.016450] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing check_logdir /tmp/testlogs/ [ 324.963619] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing yml_node [ 330.091676] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 332.845627] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 335.744822] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 337.421504] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 03:14:51 EDT 2026 [ 354.291555] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 356.004083] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 373.737639] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.746018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 373.759278] Lustre: Skipped 1 previous similar message [ 373.781875] Lustre: Skipped 3 previous similar messages [ 378.105700] Lustre: server umount lustre-MDT0000 complete [ 383.072943] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024538 with bad export cookie 4839353539222340014 [ 383.079045] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 383.090854] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 383.970311] LustreError: 12270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.973323] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.985318] LustreError: 12270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 384.019657] Lustre: Skipped 2 previous similar messages [ 384.027545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 384.037742] Lustre: Skipped 1 previous similar message [ 389.093222] LustreError: 12270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.094895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.129542] LustreError: 12270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 389.171305] Lustre: Skipped 1 previous similar message [ 389.814552] Lustre: server umount lustre-MDT0001 complete [ 395.345571] Lustre: server umount lustre-OST0000 complete [ 401.350923] Lustre: server umount lustre-OST0001 complete [ 409.412875] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 420.513510] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 461.112149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.501386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 461.544421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 461.704627] Lustre: lustre-MDT0000: new disk, initializing [ 461.810121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.864755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 467.183895] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 480.003925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.187617] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 480.272242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 480.275626] Lustre: Skipped 1 previous similar message [ 480.471915] Lustre: lustre-MDT0001: new disk, initializing [ 480.676200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 480.779437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 480.806264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 487.344593] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 498.066653] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 504.659806] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 511.646497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.012897] Lustre: lustre-OST0000: new disk, initializing [ 512.023353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 512.041427] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 512.152448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.240349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 513.265719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 513.338754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 518.908687] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 529.105152] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 534.205751] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 534.485327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 540.088878] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 540.323977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.225729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.227199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.233197] LustreError: Skipped 1 previous similar message [ 544.238068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.265906] Lustre: Skipped 2 previous similar messages [ 544.302927] Lustre: Skipped 1 previous similar message [ 548.485363] Lustre: server umount lustre-OST0000 complete [ 554.466682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.480053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.488876] Lustre: Skipped 1 previous similar message [ 554.506288] Lustre: Skipped 1 previous similar message [ 563.681139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.688299] Lustre: Skipped 3 previous similar messages [ 567.263323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.514464] Lustre: server umount lustre-MDT0000 complete [ 568.806392] LustreError: 15916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.832917] LustreError: 15916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 572.038190] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024727 with bad export cookie 4839353539222341547 [ 572.061790] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 572.065195] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.762136] Lustre: server umount lustre-MDT0001 complete [ 576.505939] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:18:50 (1789024730) [ 583.259892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.737818] LustreError: 19317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.781312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.785527] LustreError: 19318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.826890] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 599.009770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.816777] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 612.375523] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 619.090026] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 627.740546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 634.921854] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 642.803779] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 654.306451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.314066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 654.315616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 654.328226] Lustre: Skipped 1 previous similar message [ 659.202253] Lustre: server umount lustre-OST0000 complete [ 664.545435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.843506] Lustre: server umount lustre-MDT0000 complete [ 673.770378] LustreError: 20332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.791384] LustreError: 20332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 675.098436] LustreError: 20052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024830 with bad export cookie 4839353539222342429 [ 675.101800] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 675.115841] LustreError: 20052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 675.664204] Lustre: server umount lustre-MDT0001 complete [ 688.159265] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 691.580165] Key type lgssc unregistered [ 692.014475] LNet: 22854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.030040] LNetError: 22854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.054491] LNet: Removed LNI 192.168.202.150@tcp [ 693.296145] Key type .llcrypt unregistered [ 693.300209] Key type ._llcrypt unregistered [ 719.841510] Key type ._llcrypt registered [ 719.843380] Key type .llcrypt registered [ 719.928801] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 720.960888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.973399] alg: No test for adler32 (adler32-zlib) [ 722.130392] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 722.277168] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 723.895228] Key type lgssc registered [ 725.578332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 732.801787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 732.844718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 734.224264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.862656] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 744.429413] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.459362] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 749.536758] LustreError: 24267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 749.808619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.271332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 754.107234] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 761.485834] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 767.602420] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 775.035803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 775.148366] LustreError: 26086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.338670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.472968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 783.235900] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 792.451454] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 809.746262] Lustre: server umount lustre-OST0000 complete [ 815.584877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.594694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.605470] Lustre: Skipped 1 previous similar message [ 820.706594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 820.729454] Lustre: Skipped 2 previous similar messages [ 823.038835] Lustre: server umount lustre-MDT0000 complete [ 825.842374] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 825.863892] LustreError: 24266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 826.849219] Lustre: 23578:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789024966/real 1789024966] req@00000000fbe4ed1e x1875928549455232/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789024982 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 826.898065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.322354] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789024982 with bad export cookie 8488697788258467764 [ 827.322713] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.344916] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 827.819713] Lustre: server umount lustre-MDT0001 complete [ 839.147487] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 843.277777] Key type lgssc unregistered [ 843.594253] LNet: 28241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.612311] LNetError: 28241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.643294] LNet: Removed LNI 192.168.202.150@tcp [ 844.685076] Key type .llcrypt unregistered [ 844.689230] Key type ._llcrypt unregistered [ 870.010352] Key type ._llcrypt registered [ 870.012741] Key type .llcrypt registered [ 870.110106] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 871.073156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.237128] alg: No test for adler32 (adler32-zlib) [ 872.466647] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 872.773189] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 874.535388] Key type lgssc registered [ 876.224492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 885.642211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 885.669856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 887.282185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.916397] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 896.482682] LustreError: 29654:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 896.503341] LustreError: 29654:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 900.665436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.077894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.198985] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 912.277941] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 919.359841] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.295851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.711405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.614720] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 948.150680] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 957.239103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 966.625760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 966.646457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.666789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.655677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 967.657925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 967.681610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.440600] Lustre: server umount lustre-OST0000 complete [ 977.888265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 977.888591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.930953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 977.935995] Lustre: Skipped 2 previous similar messages [ 983.009217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.026173] Lustre: Skipped 1 previous similar message [ 988.135944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 988.144764] Lustre: Skipped 1 previous similar message [ 990.175349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 991.061125] Lustre: server umount lustre-MDT0000 complete [ 993.250838] LustreError: 30384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 993.270236] LustreError: 30384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 995.578149] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025151 with bad export cookie 13483721474959467824 [ 995.591698] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 995.598700] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 996.310840] Lustre: server umount lustre-MDT0001 complete [ 1008.854151] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1012.885842] Key type lgssc unregistered [ 1013.292611] LNet: 33561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.299798] LNetError: 33561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.318640] LNet: Removed LNI 192.168.202.150@tcp [ 1014.229287] Key type .llcrypt unregistered [ 1014.238938] Key type ._llcrypt unregistered [ 1043.875771] Key type ._llcrypt registered [ 1043.882605] Key type .llcrypt registered [ 1043.969776] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1045.168937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.231906] alg: No test for adler32 (adler32-zlib) [ 1046.390895] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1046.633791] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1048.303587] Key type lgssc registered [ 1049.391486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1057.740821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1057.809164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.364064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.725892] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1068.532429] LustreError: 34975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.565595] LustreError: 34975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1072.115737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.627798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.887576] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1085.682273] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1092.909455] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1099.768494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.293712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.396801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1107.115617] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1115.103411] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1124.835083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.849136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.882409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.880195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.890359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.895384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.925228] Lustre: Skipped 1 previous similar message [ 1129.456293] Lustre: server umount lustre-OST0000 complete [ 1134.050563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.057305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.070490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.057404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.073261] Lustre: Skipped 3 previous similar messages [ 1139.476525] Lustre: server umount lustre-MDT0000 complete [ 1142.239813] LustreError: 34975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.252551] LustreError: 34975:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1142.607235] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025298 with bad export cookie 6627975859096429138 [ 1142.626325] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.630051] LustreError: 34957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1143.107944] Lustre: server umount lustre-MDT0001 complete [ 1154.430664] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1157.634562] Key type lgssc unregistered [ 1157.969681] LNet: 38505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.973478] LNetError: 38505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.984644] LNet: Removed LNI 192.168.202.150@tcp [ 1158.861701] Key type .llcrypt unregistered [ 1158.863828] Key type ._llcrypt unregistered [ 1186.002676] Key type ._llcrypt registered [ 1186.009064] Key type .llcrypt registered [ 1186.182695] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1187.385081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1187.400527] alg: No test for adler32 (adler32-zlib) [ 1188.631784] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1188.956942] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1190.671198] Key type lgssc registered [ 1192.092419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1204.759856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1204.771702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.354146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.352195] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1215.522921] LustreError: 39919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.552382] LustreError: 39919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1220.587971] LustreError: 39920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.772648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.088875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1226.783357] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1235.477100] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1242.372456] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.052433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.819500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.958032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1258.537897] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1266.860228] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1275.119687] Lustre: lustre-OST0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1276.384143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1276.402471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.402908] LustreError: Skipped 1 previous similar message [ 1276.404667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.437246] Lustre: Skipped 1 previous similar message [ 1280.221126] Lustre: lustre-OST0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1280.236574] Lustre: Skipped 1 previous similar message [ 1280.580528] Lustre: server umount lustre-OST0000 complete [ 1305.058781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.061397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1305.076435] Lustre: Skipped 1 previous similar message [ 1305.100157] Lustre: Skipped 1 previous similar message [ 1310.068711] Lustre: server umount lustre-MDT0000 complete [ 1310.190586] LustreError: 39919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.215093] LustreError: 39919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1314.795642] LustreError: 39901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025470 with bad export cookie 6237629836098457195 [ 1314.802068] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.804534] LustreError: 39901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1329.119153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1329.648251] Lustre: server umount lustre-MDT0001 complete [ 1340.433722] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1343.383539] Key type lgssc unregistered [ 1343.723207] LNet: 43628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.732312] LNetError: 43628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.752934] LNet: Removed LNI 192.168.202.150@tcp [ 1344.733875] Key type .llcrypt unregistered [ 1344.735216] Key type ._llcrypt unregistered [ 1367.467665] Key type ._llcrypt registered [ 1367.473330] Key type .llcrypt registered [ 1367.635410] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1368.472125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1368.567663] alg: No test for adler32 (adler32-zlib) [ 1369.878954] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1370.120668] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1371.791286] Key type lgssc registered [ 1372.876938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1381.098819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1381.133503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.467815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.578514] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1391.650657] LustreError: 45037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.675673] LustreError: 45037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1395.768623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.323711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.579410] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1409.308303] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1416.662590] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1424.120451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.569362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.643779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1433.072284] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1441.858203] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1449.714653] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1451.492228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.498920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1451.512884] Lustre: Skipped 1 previous similar message [ 1451.524715] Lustre: Skipped 2 previous similar messages [ 1454.792876] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.50@tcp (stopping) [ 1455.826436] Lustre: server umount lustre-MDT0000 complete [ 1456.614923] LustreError: 45032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.628295] LustreError: 45032:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1459.543795] LustreError: 45018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789025615 with bad export cookie 5324621180750569808 [ 1459.553040] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.569248] LustreError: 45018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1459.922365] LustreError: 45033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1459.961768] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.50@tcp (stopping) [ 1461.735576] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.759941] Lustre: Skipped 1 previous similar message [ 1464.739039] LustreError: 46679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.739307] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1464.768576] LustreError: 46679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1464.779428] Lustre: Skipped 1 previous similar message [ 1466.123595] Lustre: server umount lustre-MDT0001 complete [ 1496.031157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1496.327679] Lustre: server umount lustre-OST0000 complete [ 1515.454231] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1518.655774] Key type lgssc unregistered [ 1518.946128] LNet: 48901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.954082] LNetError: 48901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.978643] LNet: Removed LNI 192.168.202.150@tcp [ 1519.785834] Key type .llcrypt unregistered [ 1519.788130] Key type ._llcrypt unregistered [ 1547.195256] Key type ._llcrypt registered [ 1547.198647] Key type .llcrypt registered [ 1547.344518] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1549.249701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1549.343722] alg: No test for adler32 (adler32-zlib) [ 1550.530495] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1550.861620] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1552.735252] Key type lgssc registered [ 1554.377428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.648078] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1565.675557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1581.281454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1586.399648] LustreError: 50294:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 1613.088485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.653609] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1627.853829] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1792.649803] Lustre: server umount lustre-OST0000 complete [ 1809.776653] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1813.048812] Key type lgssc unregistered [ 1813.325585] LNet: 51970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.333043] LNetError: 51970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.350709] LNet: Removed LNI 192.168.202.150@tcp [ 1814.162946] Key type .llcrypt unregistered [ 1814.172966] Key type ._llcrypt unregistered [ 1844.984928] Key type ._llcrypt registered [ 1844.987260] Key type .llcrypt registered [ 1845.220856] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1847.746543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.819744] alg: No test for adler32 (adler32-zlib) [ 1849.165367] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1849.754811] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1851.571207] Key type lgssc registered [ 1853.631732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1862.989779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1863.000410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.512246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.858230] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1874.918978] LustreError: 53382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.945124] LustreError: 53382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1878.975967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.437128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.702104] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1893.645218] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.311509] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1910.439311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1911.011408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.769949] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 1927.202650] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.466294] Lustre: server umount lustre-OST0000 complete [ 1934.819753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.832667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.835258] Lustre: Skipped 1 previous similar message [ 1934.846084] Lustre: Skipped 1 previous similar message [ 1939.938258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.059974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.069795] Lustre: Skipped 1 previous similar message [ 1949.151721] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1949.318366] Lustre: server umount lustre-MDT0000 complete [ 1950.184346] LustreError: 53381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.228872] LustreError: 53381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1951.199587] Lustre: 52690:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789026090/real 1789026090] req@00000000c6545eff x1875929732246144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789026106 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1951.232810] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.242788] LustreError: 53382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.272935] LustreError: 53382:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1952.822534] LustreError: 53363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026108 with bad export cookie 16566971801194176278 [ 1952.834809] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.845920] LustreError: 53363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1953.152511] Lustre: server umount lustre-MDT0001 complete [ 1963.674220] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 1966.496593] Key type lgssc unregistered [ 1966.820464] LNet: 56912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.827809] LNetError: 56912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.845548] LNet: Removed LNI 192.168.202.150@tcp [ 1967.576106] Key type .llcrypt unregistered [ 1967.578140] Key type ._llcrypt unregistered [ 1992.210308] Key type ._llcrypt registered [ 1992.212573] Key type .llcrypt registered [ 1992.328356] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 1993.474074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.518395] alg: No test for adler32 (adler32-zlib) [ 1995.355756] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 1995.964052] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 1997.639252] Key type lgssc registered [ 1999.316999] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2007.930849] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2007.943052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2023.520569] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2028.639390] LustreError: 58294:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 2055.263867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.100994] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2069.962979] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.709297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.513598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.577399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2082.266257] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2082.791391] LustreError: 58320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.808622] LustreError: 58320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2087.906031] LustreError: 58319:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.979769] LustreError: 58320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.817820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.369170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2096.763490] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2104.560328] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2110.382794] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.006737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2113.010105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.018927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2113.034352] Lustre: Skipped 1 previous similar message [ 2113.059970] Lustre: Skipped 1 previous similar message [ 2118.120651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2118.130874] Lustre: Skipped 1 previous similar message [ 2118.436731] Lustre: server umount lustre-OST0000 complete [ 2143.215443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2143.227170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.240042] Lustre: Skipped 1 previous similar message [ 2143.260522] Lustre: Skipped 1 previous similar message [ 2146.684588] Lustre: server umount lustre-MDT0000 complete [ 2148.330903] LustreError: 59336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.349990] LustreError: 59336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2150.391684] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026305 with bad export cookie 5217555902087483107 [ 2150.414380] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.415660] LustreError: 58300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2156.852649] Lustre: server umount lustre-MDT0001 complete [ 2167.095500] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2169.931745] Key type lgssc unregistered [ 2170.217647] LNet: 62038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.226988] LNetError: 62038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.242634] LNet: Removed LNI 192.168.202.150@tcp [ 2171.229821] Key type .llcrypt unregistered [ 2171.232099] Key type ._llcrypt unregistered [ 2193.204283] Key type ._llcrypt registered [ 2193.206568] Key type .llcrypt registered [ 2193.309580] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2194.370781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2194.504813] alg: No test for adler32 (adler32-zlib) [ 2195.607335] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2195.927843] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2197.647307] Key type lgssc registered [ 2199.027226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.754887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2206.778269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.345189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.798192] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2218.465477] LustreError: 63449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2218.495255] LustreError: 63449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2221.945601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.693573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2228.435929] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2236.933782] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2242.910680] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2250.420259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2250.852704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2253.956924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2256.736227] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2264.715899] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2284.009054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.021160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.124463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.147522] Lustre: Skipped 2 previous similar messages [ 2294.243715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.256242] Lustre: Skipped 1 previous similar message [ 2297.311153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2297.600212] Lustre: server umount lustre-OST0000 complete [ 2304.482781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2304.493465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.497599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.519304] Lustre: Skipped 2 previous similar messages [ 2307.499281] Lustre: server umount lustre-MDT0000 complete [ 2309.608501] LustreError: 63448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.641752] LustreError: 63448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2311.374862] LustreError: 63431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026466 with bad export cookie 13935173242344024374 [ 2311.406038] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2311.884630] Lustre: server umount lustre-MDT0001 complete [ 2322.966120] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2325.560881] Key type lgssc unregistered [ 2325.954433] LNet: 67006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.958797] LNetError: 67006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.974342] LNet: Removed LNI 192.168.202.150@tcp [ 2326.822660] Key type .llcrypt unregistered [ 2326.825046] Key type ._llcrypt unregistered [ 2360.986853] Key type ._llcrypt registered [ 2361.000655] Key type .llcrypt registered [ 2361.227993] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2362.848192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.995155] alg: No test for adler32 (adler32-zlib) [ 2364.342221] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2364.826473] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2366.689419] Key type lgssc registered [ 2367.926758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2377.745498] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 03:48:51 (1789026531) [ 2385.340877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2385.360943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.055094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.154489] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2396.660597] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.681598] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2401.766806] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.073243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.692727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2407.946929] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2415.899338] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2422.280815] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2428.944947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2429.271809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2432.387790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2434.595118] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2441.821279] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2450.456796] Lustre: Failing over lustre-MDT0000 [ 2450.737391] Lustre: server umount lustre-MDT0000 complete [ 2452.967710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.974580] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.977562] Lustre: Skipped 2 previous similar messages [ 2453.010275] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2458.087468] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.112321] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2458.964993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.136563] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.345837] Lustre: *** cfs_fail_loc=135, val=0*** [ 2459.485662] LustreError: 71792:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2459.494574] LustreError: 71792:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.150@tcp: cfg command failed: rc = -2 [ 2459.503634] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2459.503634] [ 2459.514266] LustreError: MGC192.168.202.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2459.525781] LustreError: 71780:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2459.532442] LustreError: 71780:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2459.538957] LustreError: 71780:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2459.544438] LustreError: 71780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2459.567196] Lustre: server umount lustre-MDT0000 complete [ 2459.570694] LustreError: 71780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2463.205322] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.231887] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2466.857779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.983960] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2467.009929] LustreError: 68699:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2467.353870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.440777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2469.082325] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2472.428605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2472.484711] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2472.563701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2473.204512] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2482.663141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.679673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.126387] Lustre: server umount lustre-OST0000 complete [ 2492.896746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.904712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.916524] Lustre: Skipped 1 previous similar message [ 2492.928692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.947316] Lustre: Skipped 2 previous similar messages [ 2498.025352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.139211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.147114] Lustre: Skipped 2 previous similar messages [ 2505.184084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2505.707836] Lustre: server umount lustre-MDT0000 complete [ 2508.259028] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.284477] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2509.821282] LustreError: 68688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026665 with bad export cookie 50229064228086319 [ 2509.828498] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.863862] LustreError: 68688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2510.367572] Lustre: server umount lustre-MDT0001 complete [ 2524.389515] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2528.481383] Key type lgssc unregistered [ 2528.821343] LNet: 73575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2528.832453] LNetError: 73575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2528.863496] LNet: Removed LNI 192.168.202.150@tcp [ 2530.136424] Key type .llcrypt unregistered [ 2530.141926] Key type ._llcrypt unregistered [ 2554.108239] Key type ._llcrypt registered [ 2554.109880] Key type .llcrypt registered [ 2554.223564] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2555.663271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2555.697421] alg: No test for adler32 (adler32-zlib) [ 2556.850751] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2557.170051] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2558.872365] Key type lgssc registered [ 2560.104942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2568.635763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2568.656706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.103759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.259568] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2580.452020] LustreError: 74982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.478176] LustreError: 74982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2585.586201] LustreError: 74983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.879105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.533264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.118312] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2603.892980] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2611.426306] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2619.113475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2619.559953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.014483] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2637.411974] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2640.074103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2648.373449] Lustre: Failing over lustre-MDT0000 [ 2648.714933] Lustre: server umount lustre-MDT0000 complete [ 2649.848688] LustreError: 74979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.595557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.614257] Lustre: Skipped 1 previous similar message [ 2654.939987] LustreError: 74978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.956586] LustreError: 74978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2659.290719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.457891] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.633371] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2659.784804] LustreError: 78072:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2659.789909] LustreError: 78072:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.150@tcp: cfg command failed: rc = -9 [ 2659.800678] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2659.800678] [ 2659.812169] LustreError: MGC192.168.202.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2659.825320] LustreError: 78061:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2659.832256] LustreError: 78061:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2659.841502] LustreError: 78061:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2659.847392] LustreError: 78061:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2659.867197] Lustre: server umount lustre-MDT0000 complete [ 2659.871572] LustreError: 78061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2660.046443] LustreError: 76030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.068512] LustreError: 76030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2667.683132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.842770] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.852927] LustreError: 74975:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2668.269410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.375831] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2671.821423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2673.657443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2673.705375] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2673.760224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2674.211920] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2683.879183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.895174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.900930] Lustre: Skipped 2 previous similar messages [ 2689.008179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.023979] Lustre: Skipped 2 previous similar messages [ 2690.312923] Lustre: server umount lustre-OST0000 complete [ 2699.233276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.237538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2699.246063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.247072] Lustre: Skipped 1 previous similar message [ 2701.218377] Lustre: server umount lustre-MDT0000 complete [ 2704.358737] LustreError: 74979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.407183] LustreError: 74979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 2704.804543] LustreError: 74963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789026860 with bad export cookie 15707497868062603647 [ 2704.819466] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.577920] Lustre: server umount lustre-MDT0001 complete [ 2718.050879] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2720.780749] Key type lgssc unregistered [ 2721.066639] LNet: 79853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.073118] LNetError: 79853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.089190] LNet: Removed LNI 192.168.202.150@tcp [ 2722.061159] Key type .llcrypt unregistered [ 2722.062592] Key type ._llcrypt unregistered [ 2734.678953] Key type ._llcrypt registered [ 2734.682276] Key type .llcrypt registered [ 2734.843819] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2735.103769] Key type .llcrypt unregistered [ 2735.105537] Key type ._llcrypt unregistered [ 2759.001140] Key type ._llcrypt registered [ 2759.006712] Key type .llcrypt registered [ 2759.200109] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2760.539581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.665277] alg: No test for adler32 (adler32-zlib) [ 2761.735720] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2761.947187] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2763.599224] Key type lgssc registered [ 2764.826696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2772.359085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2772.377676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.004869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.175557] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2784.224486] LustreError: 81749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.251454] LustreError: 81749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2786.365391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.753695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2791.580773] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2799.170435] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2803.893695] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2810.409740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.677959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.726141] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2818.869652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2823.484903] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2834.407489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.424710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.425955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2834.447623] Lustre: Skipped 1 previous similar message [ 2839.281290] Lustre: server umount lustre-OST0000 complete [ 2843.110550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2843.122859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.133513] Lustre: Skipped 1 previous similar message [ 2843.138444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.644456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.647073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.670246] Lustre: Skipped 1 previous similar message [ 2849.241324] Lustre: server umount lustre-MDT0000 complete [ 2849.761736] LustreError: 81749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.784910] LustreError: 81749:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2852.672174] LustreError: 82485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027008 with bad export cookie 9417358829959166316 [ 2852.678931] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2852.680197] LustreError: 82485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2853.106569] Lustre: server umount lustre-MDT0001 complete [ 2865.018652] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 2867.953347] Key type lgssc unregistered [ 2868.230784] LNet: 85281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.240288] LNetError: 85281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2868.251719] LNet: Removed LNI 192.168.202.150@tcp [ 2869.219230] Key type .llcrypt unregistered [ 2869.222130] Key type ._llcrypt unregistered [ 2893.690665] Key type ._llcrypt registered [ 2893.693664] Key type .llcrypt registered [ 2893.834886] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 2894.906456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2894.930903] alg: No test for adler32 (adler32-zlib) [ 2896.070975] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 2896.369491] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 2898.104237] Key type lgssc registered [ 2899.681839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2909.843642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2909.874790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.623554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.079512] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2921.961451] LustreError: 86692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2922.005408] LustreError: 86692:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2927.088921] LustreError: 86693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.157677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.545495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.808909] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2941.852952] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2947.930564] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2955.761306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.381377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.391887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2963.537693] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 2970.270791] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2982.881737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.897053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.922885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.910447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.918538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.929555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.960874] Lustre: Skipped 1 previous similar message [ 2985.423125] Lustre: server umount lustre-OST0000 complete [ 2990.049427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2990.063503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.080229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.153171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.155133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.091369] Lustre: server umount lustre-MDT0000 complete [ 2999.270637] LustreError: 86692:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.778180] LustreError: 86675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027155 with bad export cookie 3660326135483638616 [ 2999.801972] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.803182] LustreError: 86675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3000.272288] Lustre: server umount lustre-MDT0001 complete [ 3013.122238] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3016.083820] Key type lgssc unregistered [ 3016.493786] LNet: 90225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.510314] LNetError: 90225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.533868] LNet: Removed LNI 192.168.202.150@tcp [ 3017.789701] Key type .llcrypt unregistered [ 3017.792243] Key type ._llcrypt unregistered [ 3045.911893] Key type ._llcrypt registered [ 3045.920239] Key type .llcrypt registered [ 3046.052934] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3047.335058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3047.400499] alg: No test for adler32 (adler32-zlib) [ 3048.722057] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3048.957088] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3050.623250] Key type lgssc registered [ 3051.680101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3059.893569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3059.917608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3061.526316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.313187] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3071.975768] LustreError: 91635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.003651] LustreError: 91635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3077.089766] LustreError: 91634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3079.193305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.800436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3085.790402] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3095.406746] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.026786] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3109.565102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.921400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.345903] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3123.438376] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3132.471613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3145.190083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3145.198271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.219143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3145.696339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.706807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.726042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.749678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3150.255289] Lustre: server umount lustre-OST0000 complete [ 3156.450167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3156.457051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3156.470268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.474178] Lustre: Skipped 1 previous similar message [ 3160.317822] Lustre: server umount lustre-MDT0000 complete [ 3163.115584] LustreError: 91629:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.149770] LustreError: 91629:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3164.437380] LustreError: 91616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027319 with bad export cookie 3482292709148058984 [ 3164.447979] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.705627] Lustre: server umount lustre-MDT0001 complete [ 3176.205598] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3179.727498] Key type lgssc unregistered [ 3180.104411] LNet: 95203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.122594] LNetError: 95203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.149857] LNet: Removed LNI 192.168.202.150@tcp [ 3181.160267] Key type .llcrypt unregistered [ 3181.165355] Key type ._llcrypt unregistered [ 3207.229323] Key type ._llcrypt registered [ 3207.231281] Key type .llcrypt registered [ 3207.391393] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3208.860699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3208.920884] alg: No test for adler32 (adler32-zlib) [ 3209.974480] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3210.232124] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3211.991406] Key type lgssc registered [ 3213.853747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3222.168160] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3222.192879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.791551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3242.911498] LustreError: 96586:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 3269.602467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.693548] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3284.028164] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.035575] Lustre: server umount lustre-OST0000 complete [ 3298.003869] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:04:12 (1789027452) [ 3305.192104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3305.684831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.639420] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3311.109960] LustreError: 98226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.139734] LustreError: 98226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3316.218461] LustreError: 98227:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.285104] LustreError: 98226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.360426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.761104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3324.480440] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3332.239576] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3337.659046] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3344.527104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3344.916445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.767858] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3352.150503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3359.902477] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3371.488945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3371.512580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.555648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.512306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.518953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.474733] Lustre: server umount lustre-OST0000 complete [ 3382.753391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.766059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.778561] Lustre: Skipped 1 previous similar message [ 3382.791459] Lustre: Skipped 2 previous similar messages [ 3387.872349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.893321] Lustre: Skipped 1 previous similar message [ 3388.347963] Lustre: server umount lustre-MDT0000 complete [ 3392.996399] LustreError: 98222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.018459] LustreError: 98222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3393.234860] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027548 with bad export cookie 14530579040296290432 [ 3393.245266] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3393.245829] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3393.646151] Lustre: server umount lustre-MDT0001 complete [ 3406.297232] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3409.758995] Key type lgssc unregistered [ 3410.116379] LNet: 101919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3410.136683] LNetError: 101919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3410.171530] LNet: Removed LNI 192.168.202.150@tcp [ 3411.242131] Key type .llcrypt unregistered [ 3411.245151] Key type ._llcrypt unregistered [ 3456.510627] Key type ._llcrypt registered [ 3456.512283] Key type .llcrypt registered [ 3456.610873] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3458.034166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3458.146761] alg: No test for adler32 (adler32-zlib) [ 3459.184216] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3459.484760] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3461.255893] Key type lgssc registered [ 3462.455923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3471.102801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3471.154368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3472.881618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.617210] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3483.112387] LustreError: 104102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.141354] LustreError: 104102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3487.860209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.298992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3493.394602] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3501.520241] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3507.948439] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.401205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.782628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3518.846752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3523.757455] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3531.797924] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3544.547733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.550844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3544.577728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3544.591049] LustreError: Skipped 1 previous similar message [ 3544.614549] Lustre: Skipped 1 previous similar message [ 3549.686881] LustreError: 105925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3549.711307] LustreError: 105925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3549.974794] Lustre: server umount lustre-OST0000 complete [ 3554.788919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.803230] Lustre: Skipped 1 previous similar message [ 3554.808912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.905598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.937088] Lustre: Skipped 2 previous similar messages [ 3560.264291] LustreError: 106885:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000aff381da) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3560.691371] Lustre: server umount lustre-MDT0000 complete [ 3564.976583] LustreError: 104082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789027720 with bad export cookie 16655264481623916708 [ 3564.983888] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3565.000674] LustreError: 104082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3565.308878] Lustre: server umount lustre-MDT0001 complete [ 3577.824559] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 3581.229421] Key type lgssc unregistered [ 3581.528298] LNet: 107638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.538337] LNetError: 107638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.554965] LNet: Removed LNI 192.168.202.150@tcp [ 3582.640703] Key type .llcrypt unregistered [ 3582.642812] Key type ._llcrypt unregistered [ 3604.768632] Key type ._llcrypt registered [ 3604.771427] Key type .llcrypt registered [ 3604.898790] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3606.678252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3606.750195] alg: No test for adler32 (adler32-zlib) [ 3607.927914] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 3608.301117] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 3610.079988] Key type lgssc registered [ 3611.514865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3620.593588] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3620.615723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3636.260742] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3641.375472] LustreError: 108881:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 3668.191646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3675.165646] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3684.222818] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3691.809809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.109524] LustreError: MGC192.168.202.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3692.117360] LustreError: 109905:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3692.121770] LustreError: 109905:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3692.126952] LustreError: 109905:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3692.253889] Lustre: server umount lustre-MDT0000 complete [ 3692.269423] LustreError: 109905:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3712.479253] Lustre: 108217:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789027851/real 1789027851] req@000000000f07a998 x1875931575618944/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789027867 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3712.479818] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.520083] Lustre: 108217:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3712.559321] LustreError: 108893:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3733.062839] Lustre: server umount lustre-OST0000 complete [ 3744.184486] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 3754.372328] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3790.229574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.525165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3790.548159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3790.628552] Lustre: lustre-MDT0000: new disk, initializing [ 3790.737885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.771307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3795.620346] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3807.851210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.012538] Lustre: 112700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3808.095295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3808.100355] Lustre: Skipped 1 previous similar message [ 3808.244848] Lustre: lustre-MDT0001: new disk, initializing [ 3808.488879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.588282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3808.597149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3813.560347] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3824.297635] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.407705] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3837.950296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3838.261551] Lustre: lustre-OST0000: new disk, initializing [ 3838.265173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3838.273444] Lustre: 114668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3838.382254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3843.980608] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3847.854701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3847.863277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3848.049080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3853.742125] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3859.254639] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3859.489480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3863.747251] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3863.896854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3868.641357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3868.653479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.655546] LustreError: Skipped 1 previous similar message [ 3868.656159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3868.687223] Lustre: Skipped 1 previous similar message [ 3871.759913] Lustre: server umount lustre-OST0000 complete [ 3878.883935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.901469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3878.904090] Lustre: Skipped 1 previous similar message [ 3881.980324] Lustre: server umount lustre-MDT0000 complete [ 3885.959182] LustreError: 112692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028041 with bad export cookie 1255201557246431617 [ 3885.964447] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3885.977326] LustreError: 112692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3886.393667] Lustre: server umount lustre-MDT0001 complete [ 3895.017912] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:14:09 (1789028049) [ 3898.972338] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3916.073785] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 3928.553346] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 3967.132039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.345367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3967.372576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3967.487885] Lustre: lustre-MDT0000: new disk, initializing [ 3967.615399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.637661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3972.591813] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3984.225723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3984.411695] Lustre: 119571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3984.482281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3984.486709] Lustre: Skipped 1 previous similar message [ 3984.586049] Lustre: lustre-MDT0001: new disk, initializing [ 3984.711639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3984.765207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3984.776692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3988.747863] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 3998.328832] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4004.333287] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4011.674615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4011.917215] Lustre: lustre-OST0000: new disk, initializing [ 4011.920632] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4011.925968] Lustre: 121539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4012.008627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4018.328658] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4021.367108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4021.384958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4021.492633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4027.477414] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4032.100146] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4032.351352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4036.820140] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4036.981531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4042.209586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4042.213622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.225409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4042.243976] Lustre: Skipped 2 previous similar messages [ 4047.337753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4047.348737] Lustre: Skipped 2 previous similar messages [ 4052.466249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4052.476342] Lustre: Skipped 1 previous similar message [ 4052.960918] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4053.374364] Lustre: server umount lustre-OST0000 complete [ 4057.575270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.690633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4062.693437] Lustre: Skipped 2 previous similar messages [ 4063.194480] Lustre: server umount lustre-MDT0000 complete [ 4067.743886] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028223 with bad export cookie 1255201557246432499 [ 4067.766211] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.782727] LustreError: 119562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4068.224063] Lustre: server umount lustre-MDT0001 complete [ 4077.996368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.795650] LustreError: 123754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.821483] LustreError: 123754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4078.927708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.114925] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4084.205057] LustreError: 123755:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4089.312855] LustreError: 123754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.738868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.298276] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4107.678501] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4113.553180] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4120.736924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.833618] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4134.522770] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4145.230926] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4149.729474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.733634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.751932] LustreError: Skipped 1 previous similar message [ 4149.757765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4149.775767] Lustre: Skipped 2 previous similar messages [ 4149.798774] Lustre: Skipped 1 previous similar message [ 4155.409779] Lustre: server umount lustre-OST0000 complete [ 4159.969041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.999695] Lustre: Skipped 1 previous similar message [ 4173.279162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4175.329614] LustreError: 123754:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4175.345834] LustreError: 123754:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4175.436073] Lustre: server umount lustre-MDT0000 complete [ 4178.641835] LustreError: 123736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028334 with bad export cookie 1255201557246433381 [ 4178.647931] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.651623] LustreError: 123736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4178.863386] Lustre: server umount lustre-MDT0001 complete [ 4188.107767] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 4190.603280] Key type lgssc unregistered [ 4190.813203] LNet: 127385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4190.825915] LNetError: 127385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4190.853600] LNet: Removed LNI 192.168.202.150@tcp [ 4191.464502] Key type .llcrypt unregistered [ 4191.469668] Key type ._llcrypt unregistered [ 4207.406094] Key type ._llcrypt registered [ 4207.409909] Key type .llcrypt registered [ 4207.504151] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 4221.233866] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 4222.244637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4222.300440] alg: No test for adler32 (adler32-zlib) [ 4223.305420] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 4223.527544] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 4225.159264] Key type lgssc registered [ 4226.114739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4258.659568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4258.723237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4259.995577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4260.017869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4260.133257] Lustre: lustre-MDT0000: new disk, initializing [ 4260.245644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.279071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4265.403082] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4278.545981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.638321] Lustre: 130531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4278.688391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4278.691941] Lustre: Skipped 1 previous similar message [ 4278.755410] Lustre: lustre-MDT0001: new disk, initializing [ 4278.845483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4278.876719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4278.887289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4284.108563] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4295.197580] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.270477] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4308.848089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4309.087473] Lustre: lustre-OST0000: new disk, initializing [ 4309.091877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4309.097644] Lustre: 132499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4309.203786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.436810] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4318.367245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4318.391877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4318.527372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4326.163570] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4331.680270] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4332.049931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4336.452186] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4336.617362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4339.184381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.185953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.191950] LustreError: Skipped 1 previous similar message [ 4339.194662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.230942] Lustre: Skipped 1 previous similar message [ 4344.295480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4344.300973] Lustre: Skipped 2 previous similar messages [ 4349.420685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.431833] Lustre: Skipped 1 previous similar message [ 4352.992176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4353.205593] Lustre: server umount lustre-OST0000 complete [ 4359.650237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4359.651668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4359.677432] Lustre: Skipped 1 previous similar message [ 4359.695585] Lustre: Skipped 1 previous similar message [ 4364.175822] Lustre: server umount lustre-MDT0000 complete [ 4367.555727] LustreError: 130523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028523 with bad export cookie 14569419929979552307 [ 4367.564135] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.564350] LustreError: 130523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4367.849142] Lustre: server umount lustre-MDT0001 complete [ 4375.620798] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 04:22:09 (1789028529) [ 4382.651430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4383.239928] LustreError: 134914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.271477] LustreError: 134914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4383.359299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4388.098194] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4388.844479] LustreError: 134915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.930928] LustreError: 134914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.123075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.686112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4403.676856] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4411.854312] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4417.663499] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4419.599969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4419.623827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.640669] Lustre: Skipped 1 previous similar message [ 4426.022546] Lustre: server umount lustre-MDT0000 complete [ 4428.256795] LustreError: 134910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.282780] LustreError: 134910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4429.689165] LustreError: 134895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028585 with bad export cookie 14569419929979553182 [ 4429.692428] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4429.701478] LustreError: 134895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4429.964872] Lustre: server umount lustre-MDT0001 complete [ 4437.140373] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 04:23:11 (1789028591) [ 4444.576239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.193939] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4465.311606] LustreError: 137329:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 4491.106143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.233864] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4504.558241] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.792765] Lustre: server umount lustre-OST0000 complete [ 4515.082643] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 04:24:28 (1789028668) [ 4523.054641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.643358] LustreError: 138775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.709906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.180111] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4534.241772] LustreError: 138776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.272817] LustreError: 138776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4540.243312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.591692] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4552.889471] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4558.385490] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4565.457859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4565.948504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.958069] Lustre: Skipped 1 previous similar message [ 4572.918709] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4580.914585] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.099174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4592.112416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.155263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.174482] Lustre: Skipped 3 previous similar messages [ 4597.622916] Lustre: server umount lustre-MDT0000 complete [ 4599.780140] LustreError: 138776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.812967] LustreError: 138776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4601.511306] LustreError: 138756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028757 with bad export cookie 14569419929979553854 [ 4601.513285] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.526256] LustreError: 138756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4601.801605] Lustre: server umount lustre-MDT0001 complete [ 4616.059783] Lustre: server umount lustre-OST0000 complete [ 4624.688620] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 04:26:18 (1789028778) [ 4632.247390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4632.594458] LustreError: 142428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4632.702495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4638.032498] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4648.661284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.994651] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4666.028233] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4673.038933] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4680.701360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.886848] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4700.307172] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4702.602959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4706.680309] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4706.988306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4712.145320] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4712.437878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4714.976373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.992298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.016642] Lustre: Skipped 2 previous similar messages [ 4715.027252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4715.036835] Lustre: Skipped 3 previous similar messages [ 4720.868077] Lustre: server umount lustre-OST0000 complete [ 4726.241320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4726.250392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.274085] Lustre: Skipped 1 previous similar message [ 4731.085060] Lustre: server umount lustre-MDT0000 complete [ 4733.412258] LustreError: 143158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.438613] LustreError: 143158:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4735.859641] LustreError: 142409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789028891 with bad export cookie 14569419929979554918 [ 4735.865318] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4735.876302] LustreError: 142409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4736.240610] Lustre: server umount lustre-MDT0001 complete [ 4745.894962] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 04:28:19 (1789028899) [ 4752.874914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.607686] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4773.730816] LustreError: 146597:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 4799.519318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.530849] Lustre: Skipped 2 previous similar messages [ 4807.081881] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4817.153124] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4827.505287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4828.665380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4832.229361] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4843.077991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4848.244569] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4856.688700] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4861.872524] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4867.452623] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4867.706735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4872.683523] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4872.941569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.331032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4879.342741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4879.359358] Lustre: Skipped 1 previous similar message [ 4879.371317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4879.373706] Lustre: Skipped 5 previous similar messages [ 4881.926345] Lustre: server umount lustre-MDT0000 complete [ 4884.962512] LustreError: 148183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.990655] LustreError: 148183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4886.010890] LustreError: 146605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029041 with bad export cookie 14569419929979555947 [ 4886.021540] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.395489] Lustre: server umount lustre-MDT0001 complete [ 4900.246690] Lustre: server umount lustre-OST0000 complete [ 4908.072327] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 04:31:02 (1789029062) [ 4915.428439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4931.039879] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4936.162087] LustreError: 150794:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 4969.185576] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4976.786393] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4984.113861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.588944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4989.044924] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 4998.138934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.154850] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5009.668363] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5015.091536] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5021.232967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.452414] Lustre: lustre-OST0001: new disk, initializing [ 5021.456332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5021.461574] Lustre: 153664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5026.987018] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5027.880370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5027.895496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5028.043221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5036.512515] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5040.794423] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5041.012099] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5045.715734] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5045.910833] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5048.801359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.812087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5048.823780] Lustre: Skipped 3 previous similar messages [ 5048.833020] Lustre: Skipped 3 previous similar messages [ 5054.288854] Lustre: server umount lustre-OST0000 complete [ 5059.040418] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5059.059276] LustreError: Skipped 1 previous similar message [ 5064.487899] Lustre: server umount lustre-OST0001 complete [ 5074.818721] Lustre: server umount lustre-MDT0000 complete [ 5078.650323] LustreError: 150801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029234 with bad export cookie 14569419929979556899 [ 5078.661832] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5078.666273] LustreError: 150801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5079.024630] Lustre: server umount lustre-MDT0001 complete [ 5103.476286] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 04:34:17 (1789029257) [ 5104.817559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5106.591614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5108.247454] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 04:34:22 (1789029262) [ 5114.905970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.313511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5115.405537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5115.635785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.644433] Lustre: Skipped 6 previous similar messages [ 5119.845619] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5128.113479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.325379] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5128.366216] Lustre: 157224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5133.607118] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5142.812479] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5149.642615] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5156.117510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.379336] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5162.879151] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5166.264805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5170.737869] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5176.346150] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5176.749020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5183.464170] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5183.902325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5187.041693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5187.063968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.090859] Lustre: Skipped 4 previous similar messages [ 5187.103865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5187.106601] Lustre: Skipped 10 previous similar messages [ 5201.375129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5201.523986] Lustre: server umount lustre-OST0000 complete [ 5225.178891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5226.667712] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.694928] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5226.696075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.142968] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5239.401820] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5244.961545] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5245.263526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5250.532396] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5250.809536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5257.603641] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff930cc488f800.ost_server_uuid 50 [ 5259.348522] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff930cc488f800.ost_server_uuid in FULL state after 0 sec [ 5259.586211] LustreError: 157233:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.50@tcp) returned error from blocking AST (req@00000000f217fdc0 x1875932221810432 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006873603f/0xca30fa3bf5ed9d7d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.50@tcp remote: 0xb40f0b38b7b26a54 expref: 5 pid: 157232 timeout: 5359 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5259.673607] LustreError: lustre-MDT0000: A client on nid 192.168.202.50@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5259.694113] LustreError: 157221:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.50@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006873603f/0xca30fa3bf5ed9d7d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.50@tcp remote: 0xb40f0b38b7b26a54 expref: 6 pid: 157232 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5265.904216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5265.910319] LustreError: Skipped 1 previous similar message [ 5271.332219] Lustre: server umount lustre-OST0000 complete [ 5282.677334] Lustre: server umount lustre-MDT0000 complete [ 5283.299270] LustreError: 159786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.329030] LustreError: 159786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5288.867672] LustreError: 157218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029444 with bad export cookie 14569419929979558082 [ 5288.894965] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.904296] LustreError: 157218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5295.788179] Lustre: server umount lustre-MDT0001 complete [ 5315.234195] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 5319.824728] Key type lgssc unregistered [ 5320.151820] LNet: 163362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.161268] LNetError: 163362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5321.199779] LNet: Removed LNI 192.168.202.150@tcp [ 5322.559730] Key type .llcrypt unregistered [ 5322.572420] Key type ._llcrypt unregistered [ 5354.255275] Key type ._llcrypt registered [ 5354.256931] Key type .llcrypt registered [ 5354.491714] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5356.679517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5356.779504] alg: No test for adler32 (adler32-zlib) [ 5358.164251] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 5358.543616] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 5360.391311] Key type lgssc registered [ 5362.428593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5371.345930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5371.365568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.814488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.705748] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5383.157739] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.205453] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5388.257458] LustreError: 164773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.172267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.504942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5393.953901] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5402.144864] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5408.930735] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.105085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5418.415360] LustreError: 166594:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.009359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5424.129729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5427.731618] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5441.290292] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5449.644860] Lustre: Failing over lustre-MDT0000 [ 5449.698746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.762135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.966244] Lustre: server umount lustre-MDT0000 complete [ 5454.832638] LustreError: 165796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.877378] LustreError: 165796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5459.937145] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5459.962959] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5462.419052] LustreError: 164752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.50@tcp arrived at 1789029617 with bad export cookie 9994293934881105468 [ 5470.180417] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.213897] LustreError: 164772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5471.200812] Lustre: 164083:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789029610/real 1789029610] req@00000000dc24dde6 x1875933411703040/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789029626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5471.248900] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5471.704677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.480826] LustreError: 164080:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c9a84bc8 x1875933411707904/t0(0) o250->MGC192.168.202.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5480.890456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5480.974760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5486.051405] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5486.071014] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5486.578763] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5489.386714] Lustre: lustre-MDT0000: Denying connection for new client 45ea6ab0-013a-4978-82d8-a972ded521e3 (at 192.168.202.50@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5494.475105] Lustre: lustre-MDT0000: Denying connection for new client 45ea6ab0-013a-4978-82d8-a972ded521e3 (at 192.168.202.50@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5500.897375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.908293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.926326] Lustre: Skipped 2 previous similar messages [ 5500.930578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5500.944148] Lustre: Skipped 2 previous similar messages [ 5504.611972] Lustre: server umount lustre-OST0000 complete [ 5508.711127] LustreError: 168509:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5508.717733] Lustre: 167857:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5508.723166] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5508.743605] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5508.760762] LustreError: 167857:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000231af28a x1875933411723264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5508.804338] LustreError: 167857:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5508.819527] LustreError: 167857:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5508.838295] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5509.285203] Lustre: server umount lustre-MDT0000 complete [ 5511.656348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5511.671620] Lustre: Skipped 1 previous similar message [ 5511.678855] LustreError: 166109:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.705723] LustreError: 166109:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5515.119419] LustreError: 164080:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5515.126306] LustreError: 164754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029670 with bad export cookie 9994293934881105566 [ 5515.179538] LustreError: 164754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5515.198648] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5521.835983] Lustre: server umount lustre-MDT0001 complete [ 5536.166268] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 5539.400796] Key type lgssc unregistered [ 5539.825729] LNet: 169263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5539.851385] LNetError: 169263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5539.873841] LNet: Removed LNI 192.168.202.150@tcp [ 5540.904870] Key type .llcrypt unregistered [ 5540.907207] Key type ._llcrypt unregistered [ 5571.220768] Key type ._llcrypt registered [ 5571.222862] Key type .llcrypt registered [ 5571.399092] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5573.549348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5573.601565] alg: No test for adler32 (adler32-zlib) [ 5574.694754] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 5574.916057] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 5576.631672] Key type lgssc registered [ 5578.082085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5587.303305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5587.338353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.050300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5594.930209] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5599.208616] LustreError: 170675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.230076] LustreError: 170675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5600.227449] LustreError: 170676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5605.348313] LustreError: 170675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5606.047568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.462111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.285641] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5621.120553] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5627.283299] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5636.583539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5637.128813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5647.535531] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5657.575078] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5659.653942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5669.860536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.861770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5669.877464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5674.306378] Lustre: server umount lustre-OST0000 complete [ 5680.097693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.106179] Lustre: Skipped 1 previous similar message [ 5680.115337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5680.124603] Lustre: Skipped 2 previous similar messages [ 5684.744681] Lustre: server umount lustre-MDT0000 complete [ 5685.216735] LustreError: 171404:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.248696] LustreError: 171404:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5689.554590] LustreError: 170656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789029845 with bad export cookie 10652731012142727844 [ 5689.559756] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5689.573169] LustreError: 170656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5690.033332] Lustre: server umount lustre-MDT0001 complete [ 5702.246077] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 5705.775850] Key type lgssc unregistered [ 5706.240169] LNet: 174207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5706.282505] LNetError: 174207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5707.335916] LNet: Removed LNI 192.168.202.150@tcp [ 5708.658204] Key type .llcrypt unregistered [ 5708.663141] Key type ._llcrypt unregistered [ 5744.547751] Key type ._llcrypt registered [ 5744.552412] Key type .llcrypt registered [ 5744.667488] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5745.588363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5745.645949] alg: No test for adler32 (adler32-zlib) [ 5746.861157] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 5747.086376] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 5748.767378] Key type lgssc registered [ 5750.195018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5758.668143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5758.712846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.398132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.919697] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5769.700181] LustreError: 175908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5769.724195] LustreError: 175908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5774.823584] LustreError: 175909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.887416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5777.800390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.122992] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5792.748300] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5802.688299] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5810.882896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.117014] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.660720] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5825.349946] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.555835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5839.846302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.849625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5839.870741] Lustre: Skipped 1 previous similar message [ 5839.879911] Lustre: Skipped 1 previous similar message [ 5844.965224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5844.980804] Lustre: Skipped 1 previous similar message [ 5845.822326] Lustre: server umount lustre-OST0000 complete [ 5849.575609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5849.588664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.600617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.200312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.211889] Lustre: Skipped 3 previous similar messages [ 5855.830575] Lustre: server umount lustre-MDT0000 complete [ 5860.320934] LustreError: 175908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.354566] LustreError: 175908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5861.312327] LustreError: 175888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030016 with bad export cookie 14404284473708411906 [ 5861.314561] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.323574] LustreError: 175888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5862.118699] Lustre: server umount lustre-MDT0001 complete [ 5875.726413] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 5878.445978] Key type lgssc unregistered [ 5878.705748] LNet: 179466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5878.710170] LNetError: 179466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5878.736177] LNet: Removed LNI 192.168.202.150@tcp [ 5879.557186] Key type .llcrypt unregistered [ 5879.563444] Key type ._llcrypt unregistered [ 5910.669474] Key type ._llcrypt registered [ 5910.675066] Key type .llcrypt registered [ 5910.866507] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 5912.200730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5912.233605] alg: No test for adler32 (adler32-zlib) [ 5913.450191] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 5913.774441] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 5915.455218] Key type lgssc registered [ 5917.073285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5929.948610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5930.004641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.515702] Lustre: *** cfs_fail_loc=135, val=0*** [ 5931.605125] LustreError: 180986:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5931.614183] LustreError: 180986:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.150@tcp: cfg command failed: rc = -2 [ 5931.636223] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5931.636223] [ 5931.647342] LustreError: MGC192.168.202.150@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5931.666056] LustreError: 180947:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5931.691725] LustreError: 180947:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5931.707656] LustreError: 180947:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5931.713146] LustreError: 180947:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5931.719677] LustreError: 180954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030087 with bad export cookie 4411225818962571979 [ 5931.821938] Lustre: server umount lustre-MDT0000 complete [ 5931.825656] LustreError: 180947:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5938.145960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.554725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.202716] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5948.904923] LustreError: 181356:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5948.932260] LustreError: 181356:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5953.182967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.672410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5959.431653] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 5969.156421] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5976.610778] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5983.727061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.742648] Lustre: Skipped 1 previous similar message [ 5983.744483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5983.779412] Lustre: Skipped 1 previous similar message [ 5985.574918] Lustre: server umount lustre-MDT0000 complete [ 5988.832838] LustreError: 181353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.885426] LustreError: 181353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5990.679986] LustreError: 181337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030146 with bad export cookie 4411225818962572133 [ 5990.681548] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.688153] LustreError: 181337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5991.133327] Lustre: server umount lustre-MDT0001 complete [ 6005.534850] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6009.950827] Key type lgssc unregistered [ 6010.625644] LNet: 183677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6010.641893] LNetError: 183677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6011.686330] LNet: Removed LNI 192.168.202.150@tcp [ 6012.495720] Key type .llcrypt unregistered [ 6012.499741] Key type ._llcrypt unregistered [ 6037.363151] Key type ._llcrypt registered [ 6037.367257] Key type .llcrypt registered [ 6037.481531] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6037.810338] Key type .llcrypt unregistered [ 6037.812223] Key type ._llcrypt unregistered [ 6056.179551] Key type ._llcrypt registered [ 6056.182600] Key type .llcrypt registered [ 6056.314545] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6057.177767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6057.409917] alg: No test for adler32 (adler32-zlib) [ 6058.477979] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6058.745396] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6060.415360] Key type lgssc registered [ 6061.771821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6070.781471] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6070.831346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6086.495429] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6091.616814] LustreError: 185801:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.150@tcp: failed processing log, type 4: rc = -110 [ 6118.240329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.998553] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6132.547439] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6140.428200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.059740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6141.213600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6145.451815] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6146.532177] LustreError: 185826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.548678] LustreError: 185826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6151.664992] LustreError: 185827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6152.738739] LustreError: 186840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.572041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.006345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6158.984887] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6166.888151] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6173.487508] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6176.287035] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6184.930579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.933740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6184.941042] Lustre: Skipped 1 previous similar message [ 6184.941877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6184.964841] Lustre: Skipped 1 previous similar message [ 6190.055687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6190.065613] Lustre: Skipped 1 previous similar message [ 6191.074144] Lustre: server umount lustre-OST0000 complete [ 6195.169619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6195.170556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6195.177976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.292193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6200.299632] Lustre: Skipped 2 previous similar messages [ 6201.494496] Lustre: server umount lustre-MDT0000 complete [ 6205.415537] LustreError: 186844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.452879] LustreError: 186844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6205.895821] LustreError: 188463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030361 with bad export cookie 6980645817784573620 [ 6205.911106] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.921484] LustreError: 188463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6206.595737] Lustre: server umount lustre-MDT0001 complete [ 6220.081678] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6224.262441] Key type lgssc unregistered [ 6224.691813] LNet: 189720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6224.717773] LNetError: 189720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6225.770258] LNet: Removed LNI 192.168.202.150@tcp [ 6227.017994] Key type .llcrypt unregistered [ 6227.020460] Key type ._llcrypt unregistered [ 6252.703575] Key type ._llcrypt registered [ 6252.705593] Key type .llcrypt registered [ 6252.825613] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6253.921901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6253.941624] alg: No test for adler32 (adler32-zlib) [ 6254.959723] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6255.172772] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6256.839721] Key type lgssc registered [ 6258.470150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6266.940087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6266.972183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6268.610088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6273.579769] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6279.145645] LustreError: 191130:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.175519] LustreError: 191130:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6282.511197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.093632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.193725] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6297.849698] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6304.728420] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6313.135500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6313.688672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.749910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6321.870775] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6330.455493] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6337.105573] Lustre: Failing over lustre-MDT0000 [ 6337.545969] Lustre: server umount lustre-MDT0000 complete [ 6338.304526] LustreError: 191124:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.556822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.567121] LustreError: 191857:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.577873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6339.579991] Lustre: Skipped 2 previous similar messages [ 6339.619106] LustreError: 191857:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6343.374068] LustreError: 191125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.491996] LustreError: 191125:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.507723] LustreError: 191125:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6355.809354] Lustre: 190442:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789030495/real 1789030495] req@000000002f3c97d4 x1875934351224064/t0(0) o400->MGC192.168.202.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789030511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6355.854757] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6358.280274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.731285] LustreError: 192294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6358.770304] LustreError: 192294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6367.645345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6367.752508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6368.971550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6372.860702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6372.898953] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6372.972018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6374.327172] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6379.959583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6390.250073] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6403.557312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.578563] Lustre: Skipped 1 previous similar message [ 6403.579284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6403.600604] Lustre: Skipped 1 previous similar message [ 6408.705233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.728725] Lustre: Skipped 1 previous similar message [ 6409.898919] Lustre: server umount lustre-OST0000 complete [ 6418.913950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6418.918298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.926033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.943023] Lustre: Skipped 1 previous similar message [ 6418.966311] Lustre: Skipped 1 previous similar message [ 6420.491332] Lustre: server umount lustre-MDT0000 complete [ 6424.041418] LustreError: 191857:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.067180] LustreError: 191857:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6425.550807] LustreError: 194961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030581 with bad export cookie 9044134048721636673 [ 6425.552843] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6425.567705] LustreError: 194961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6425.987895] Lustre: server umount lustre-MDT0001 complete [ 6437.616663] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6440.848835] Key type lgssc unregistered [ 6441.214529] LNet: 196035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6441.224072] LNetError: 196035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6441.236179] LNet: Removed LNI 192.168.202.150@tcp [ 6442.142963] Key type .llcrypt unregistered [ 6442.147099] Key type ._llcrypt unregistered [ 6467.345548] Key type ._llcrypt registered [ 6467.349465] Key type .llcrypt registered [ 6467.462294] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6468.571033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6468.618727] alg: No test for adler32 (adler32-zlib) [ 6469.736191] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6470.048255] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6471.745092] Key type lgssc registered [ 6473.365420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6481.707937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6481.736301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.159713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6487.898478] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6492.655635] LustreError: 197444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6492.673990] LustreError: 197444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6497.343945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.760102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6497.950985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6502.852615] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6512.738733] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6520.049790] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6526.523614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6527.049444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6528.063839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6534.640113] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6543.874825] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6546.895282] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6552.627705] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6569.560217] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6579.169125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.183400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.188911] LustreError: Skipped 1 previous similar message [ 6579.203945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6579.219535] Lustre: Skipped 1 previous similar message [ 6579.242820] Lustre: Skipped 1 previous similar message [ 6584.310048] LustreError: 199334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6584.330291] LustreError: 199334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6584.572619] Lustre: server umount lustre-OST0000 complete [ 6589.411286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.435167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6589.441607] Lustre: Skipped 1 previous similar message [ 6594.531261] LustreError: 197440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.547379] LustreError: 197440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6594.578577] Lustre: server umount lustre-MDT0000 complete [ 6598.822865] LustreError: 197425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030754 with bad export cookie 11859009039612861170 [ 6598.840116] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.841207] LustreError: 197425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6599.320834] Lustre: server umount lustre-MDT0001 complete [ 6612.900748] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6615.790663] Key type lgssc unregistered [ 6616.024605] LNet: 201315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6616.029139] LNetError: 201315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6616.043698] LNet: Removed LNI 192.168.202.150@tcp [ 6616.902579] Key type .llcrypt unregistered [ 6616.904712] Key type ._llcrypt unregistered [ 6642.563599] Key type ._llcrypt registered [ 6642.567443] Key type .llcrypt registered [ 6642.766110] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6644.865783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6644.983233] alg: No test for adler32 (adler32-zlib) [ 6646.371423] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6646.769409] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6648.479229] Key type lgssc registered [ 6650.418520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6660.869109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6660.895558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.432253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6668.402260] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6671.845637] LustreError: 202585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.870357] LustreError: 202585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6676.962332] LustreError: 202586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.682708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.062048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6683.887537] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6693.054805] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6699.505291] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6707.796853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.313178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6715.799319] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6724.484566] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6731.927298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6740.252818] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6754.598926] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6763.254070] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6770.393288] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6780.385047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6780.392931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.406700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6783.457936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.470051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.482927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.588240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.597633] Lustre: Skipped 2 previous similar messages [ 6793.715552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.723737] Lustre: Skipped 1 previous similar message [ 6794.207243] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6794.535393] Lustre: server umount lustre-OST0000 complete [ 6801.888941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6801.899308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6801.915943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6803.938337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6805.915474] Lustre: server umount lustre-MDT0000 complete [ 6809.060138] LustreError: 202585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.082639] LustreError: 202585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6810.124684] LustreError: 202567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789030965 with bad export cookie 2732399777875363247 [ 6810.132821] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6810.143451] LustreError: 202567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6810.783821] Lustre: server umount lustre-MDT0001 complete [ 6822.995176] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 6826.582463] Key type lgssc unregistered [ 6826.952682] LNet: 207493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6826.976872] LNetError: 207493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6827.001335] LNet: Removed LNI 192.168.202.150@tcp [ 6828.401917] Key type .llcrypt unregistered [ 6828.403713] Key type ._llcrypt unregistered [ 6857.853066] Key type ._llcrypt registered [ 6857.855610] Key type .llcrypt registered [ 6858.034768] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 6860.008251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6860.148431] alg: No test for adler32 (adler32-zlib) [ 6861.400195] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 6861.867713] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 6863.624199] Key type lgssc registered [ 6865.159335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6875.248712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6875.284620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.161855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.475533] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6887.403190] LustreError: 208904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.448647] LustreError: 208904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6892.514054] LustreError: 208905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6893.682167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.472028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6901.870816] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6910.888796] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6917.028897] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6924.543259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6924.988412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.173173] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 6938.806510] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6946.503091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6965.234465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6965.243525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.262213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6967.270854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6967.271355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6967.288724] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.318889] Lustre: Skipped 1 previous similar message [ 6969.888891] Lustre: server umount lustre-OST0000 complete [ 6976.481564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6976.492401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.518801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6980.825580] Lustre: server umount lustre-MDT0000 complete [ 6982.629795] LustreError: 209634:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.653521] LustreError: 209634:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6984.301250] LustreError: 208885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031139 with bad export cookie 17226903520560961611 [ 6984.303171] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6984.312188] LustreError: 208885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6984.639246] Lustre: server umount lustre-MDT0001 complete [ 6997.234222] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7000.141222] Key type lgssc unregistered [ 7000.419899] LNet: 212614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7000.426050] LNetError: 212614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7000.442358] LNet: Removed LNI 192.168.202.150@tcp [ 7001.529608] Key type .llcrypt unregistered [ 7001.531791] Key type ._llcrypt unregistered [ 7024.488634] Key type ._llcrypt registered [ 7024.490784] Key type .llcrypt registered [ 7024.572278] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7025.738548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7025.883190] alg: No test for adler32 (adler32-zlib) [ 7027.045820] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7027.312775] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7029.055506] Key type lgssc registered [ 7030.449504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7038.943658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7038.960709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.284230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7046.168978] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7049.697353] LustreError: 213878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7049.722575] LustreError: 213878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7054.819112] LustreError: 213879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.099196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.906029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7063.997713] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7072.989209] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7079.719200] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7086.749387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.275368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.185947] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7101.787541] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7109.832260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7128.550111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7128.556219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.577451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7130.592810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7130.596965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.601877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7130.643162] Lustre: Skipped 1 previous similar message [ 7133.625564] Lustre: server umount lustre-OST0000 complete [ 7139.808796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.821449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7139.833424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7143.864677] Lustre: server umount lustre-MDT0000 complete [ 7145.954440] LustreError: 213879:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7145.972792] LustreError: 213879:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7148.198918] LustreError: 213860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031303 with bad export cookie 12438888392910369235 [ 7148.209782] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.549649] Lustre: server umount lustre-MDT0001 complete [ 7161.204958] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7165.924819] Key type lgssc unregistered [ 7166.303710] LNet: 217583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7166.319690] LNetError: 217583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7167.408279] LNet: Removed LNI 192.168.202.150@tcp [ 7168.496217] Key type .llcrypt unregistered [ 7168.505079] Key type ._llcrypt unregistered [ 7195.078213] Key type ._llcrypt registered [ 7195.082432] Key type .llcrypt registered [ 7195.204671] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7196.844586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7197.153702] alg: No test for adler32 (adler32-zlib) [ 7198.151294] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7198.346757] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7199.991421] Key type lgssc registered [ 7201.483605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7211.301808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7211.334731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.113169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7219.502146] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7223.268201] LustreError: 218849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7223.304188] LustreError: 218849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7228.388601] LustreError: 218850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7232.566401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7233.120444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7238.133268] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7245.835632] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7252.747860] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7260.533472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.242430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7268.036262] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7275.926674] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7283.779958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7296.706980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7296.806821] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7297.073706] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7304.839871] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7313.189470] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7314.944603] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7326.931265] Lustre: Permanently deactivating lustre-OST0001 [ 7326.946392] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7332.972287] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7345.240960] Lustre: Permanently reactivating lustre-OST0001 [ 7345.257539] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7345.264454] Lustre: Skipped 2 previous similar messages [ 7351.921669] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.933621] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7351.941851] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7351.955658] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7352.043415] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7362.530720] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7362.537540] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.546680] LustreError: Skipped 1 previous similar message [ 7362.548174] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7362.580515] Lustre: Skipped 2 previous similar messages [ 7363.555359] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7363.561036] Lustre: Skipped 2 previous similar messages [ 7364.923103] Lustre: server umount lustre-OST0001 complete [ 7368.683688] LustreError: 220674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7368.707330] LustreError: 220674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7369.698556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7369.710300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7369.743956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7370.720971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7373.796262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7373.797671] LustreError: 220673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.806140] Lustre: Skipped 2 previous similar messages [ 7373.832594] LustreError: 220673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7375.857792] Lustre: server umount lustre-OST0000 complete [ 7381.472350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7381.476590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7381.496606] Lustre: Skipped 1 previous similar message [ 7381.504295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.819463] Lustre: server umount lustre-MDT0000 complete [ 7389.154357] LustreError: 218850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.172378] LustreError: 218850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7389.553261] LustreError: 218831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031545 with bad export cookie 772732279574598834 [ 7389.562510] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7389.568152] LustreError: 218831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7389.901851] Lustre: server umount lustre-MDT0001 complete [ 7401.609135] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7404.660403] Key type lgssc unregistered [ 7405.047538] LNet: 223962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.056337] LNetError: 223962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7405.077909] LNet: Removed LNI 192.168.202.150@tcp [ 7405.984524] Key type .llcrypt unregistered [ 7405.986565] Key type ._llcrypt unregistered [ 7443.521775] Key type ._llcrypt registered [ 7443.524432] Key type .llcrypt registered [ 7443.627607] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7444.910212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7444.978312] alg: No test for adler32 (adler32-zlib) [ 7446.059982] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7446.321937] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7447.999928] Key type lgssc registered [ 7449.229912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7457.553438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7457.578181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7458.942557] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7458.959521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7459.134328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.006507] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7475.231470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.451597] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7475.490838] Lustre: 226141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7475.849984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7483.407566] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7493.522833] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7501.041855] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7508.947675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7509.089586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7509.264464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.274698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7515.560738] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7523.709036] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7527.787439] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7533.112891] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7539.811610] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7547.716982] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7555.955626] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7573.096764] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7573.103894] Lustre: Skipped 1 previous similar message [ 7611.261840] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7611.269091] Lustre: Skipped 4 previous similar messages [ 7679.890260] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7679.892849] Lustre: Skipped 8 previous similar messages [ 7704.545250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7704.550925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.573077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7705.056569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7705.084845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.109535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7705.112409] Lustre: Skipped 1 previous similar message [ 7707.419169] Lustre: server umount lustre-OST0000 complete [ 7715.296394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.305546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7715.319411] Lustre: Skipped 1 previous similar message [ 7715.346624] Lustre: Skipped 1 previous similar message [ 7718.239828] Lustre: server umount lustre-MDT0000 complete [ 7722.328664] LustreError: 228005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789031877 with bad export cookie 5922701055866543767 [ 7722.339689] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.344512] LustreError: 228005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7723.179336] Lustre: server umount lustre-MDT0001 complete [ 7737.556580] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7740.128585] Key type lgssc unregistered [ 7740.546745] LNet: 230947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7740.566854] LNetError: 230947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7741.606386] LNet: Removed LNI 192.168.202.150@tcp [ 7742.993907] Key type .llcrypt unregistered [ 7742.997451] Key type ._llcrypt unregistered [ 7768.704728] Key type ._llcrypt registered [ 7768.707620] Key type .llcrypt registered [ 7768.845623] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7770.175624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7770.208086] alg: No test for adler32 (adler32-zlib) [ 7771.258528] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7771.448691] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7773.095426] Key type lgssc registered [ 7774.185790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7781.630025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7781.650839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.122940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.362203] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7792.627094] LustreError: 232357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7792.649166] LustreError: 232357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7795.499485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7795.803714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7800.570955] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7808.219260] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7813.693160] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7821.229611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7821.618177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.782388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7829.815368] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_default_debug -1 all [ 7839.206598] Lustre: DEBUG MARKER: oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7858.661833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7858.669956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.693209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7861.731800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7861.733839] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.734040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.746480] Lustre: Skipped 1 previous similar message [ 7863.541338] Lustre: server umount lustre-OST0000 complete [ 7871.970186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.982232] Lustre: Skipped 1 previous similar message [ 7871.982258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7871.997456] Lustre: Skipped 1 previous similar message [ 7874.508527] Lustre: server umount lustre-MDT0000 complete [ 7877.092513] LustreError: 233085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7877.116814] LustreError: 233085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7878.362461] LustreError: 234939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789032033 with bad export cookie 7208649518853315727 [ 7878.367832] LustreError: MGC192.168.202.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7878.381744] LustreError: 234939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7878.689347] Lustre: server umount lustre-MDT0001 complete [ 7888.314722] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 7891.134344] Key type lgssc unregistered [ 7891.465380] LNet: 236049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7891.479790] LNetError: 236049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7891.510245] LNet: Removed LNI 192.168.202.150@tcp [ 7892.472660] Key type .llcrypt unregistered [ 7892.484262] Key type ._llcrypt unregistered [ 7918.816226] Key type ._llcrypt registered [ 7918.822734] Key type .llcrypt registered [ 7919.019681] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 7920.642335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7920.662862] alg: No test for adler32 (adler32-zlib) [ 7921.830104] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 7922.041068] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 7923.703461] Key type lgssc registered [ 7925.125831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8195.802831] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8198.867836] Key type lgssc unregistered [ 8199.216815] LNet: 238068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8199.223896] LNetError: 238068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8199.247228] LNet: Removed LNI 192.168.202.150@tcp [ 8199.935296] Key type .llcrypt unregistered [ 8199.937409] Key type ._llcrypt unregistered [ 8227.910525] Key type ._llcrypt registered [ 8227.914751] Key type .llcrypt registered [ 8228.103231] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing set_hostid [ 8245.915781] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing load_modules_local [ 8246.882662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8246.921194] alg: No test for adler32 (adler32-zlib) [ 8248.167724] Lustre: Lustre: Build Version: 2.17.58_39_gf941a14 [ 8248.380288] LNet: Added LNI 192.168.202.150@tcp [8/256/0/180] [ 8250.015230] Key type lgssc registered [ 8251.084972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8285.673930] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7946 sec ========== 05:27:19 (1789032439) [ 8287.004533] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:27:21 (1789032441) === [ 8290.401555] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:27:24 (1789032444) === [ 8333.003847] Lustre: DEBUG MARKER: oleg250-server.virtnet: executing unload_modules_local [ 8336.589271] Key type lgssc unregistered [ 8336.952938] LNet: 243311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8336.957643] LNetError: 243311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8336.972279] LNet: Removed LNI 192.168.202.150@tcp [ 8337.778960] Key type .llcrypt unregistered [ 8337.780460] Key type ._llcrypt unregistered