[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 609879209 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003203] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008273] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011112] LSM: Security Framework initializing [ 0.013006] Yama: becoming mindful. [ 0.014025] SELinux: Initializing. [ 0.015068] *** VALIDATE selinux *** [ 0.023064] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027332] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] *** VALIDATE tmpfs *** [ 0.032268] *** VALIDATE proc *** [ 0.033235] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035255] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042242] debug: unmapping init [mem 0xffffffffa2659000-0xffffffffa2660fff] [ 0.044861] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045699] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048008] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050009] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053272] rcu: Hierarchical SRCU implementation. [ 0.055230] smp: Bringing up secondary CPUs ... [ 0.056528] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.060101] smp: Brought up 1 node, 4 CPUs [ 0.062024] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134613] node 0 deferred pages initialised in 69ms [ 0.138094] devtmpfs: initialized [ 0.140254] x86/mm: Memory block size: 128MB [ 0.143957] gcov: version magic: 0x41383552 [ 0.146237] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155255] pinctrl core: initialized pinctrl subsystem [ 0.158146] [ 0.159006] ************************************************************* [ 0.162010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.168009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171010] ** ** [ 0.174009] ** This means that this kernel is built to expose internal ** [ 0.176011] ** IOMMU data structures, which may compromise security on ** [ 0.180010] ** your system. ** [ 0.182007] ** ** [ 0.185009] ** If you see this message and you are not debugging the ** [ 0.188009] ** kernel, report this immediately to your vendor! ** [ 0.191009] ** ** [ 0.193007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196008] ************************************************************* [ 0.198699] NET: Registered protocol family 16 [ 0.201356] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205116] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.213180] cpuidle: using governor menu [ 0.215000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.218570] PCI: Using configuration type 1 for base access [ 0.220104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230054] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.231015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233093] cryptd: max_cpu_qlen set to 1000 [ 0.237312] ACPI: Added _OSI(Module Device) [ 0.239010] ACPI: Added _OSI(Processor Device) [ 0.241008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248987] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256402] ACPI: Interpreter enabled [ 0.259060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261025] ACPI: Using IOAPIC for interrupt routing [ 0.263109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268495] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279215] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291000] acpiphp: Slot [2] registered [ 0.292092] acpiphp: Slot [5] registered [ 0.294094] acpiphp: Slot [6] registered [ 0.296087] acpiphp: Slot [7] registered [ 0.298100] acpiphp: Slot [8] registered [ 0.300071] acpiphp: Slot [9] registered [ 0.302125] acpiphp: Slot [10] registered [ 0.304084] acpiphp: Slot [3] registered [ 0.305075] acpiphp: Slot [4] registered [ 0.307087] acpiphp: Slot [11] registered [ 0.309092] acpiphp: Slot [12] registered [ 0.310079] acpiphp: Slot [13] registered [ 0.312097] acpiphp: Slot [14] registered [ 0.314078] acpiphp: Slot [15] registered [ 0.315063] acpiphp: Slot [16] registered [ 0.317111] acpiphp: Slot [17] registered [ 0.319091] acpiphp: Slot [18] registered [ 0.321098] acpiphp: Slot [19] registered [ 0.322074] acpiphp: Slot [20] registered [ 0.323071] acpiphp: Slot [21] registered [ 0.325068] acpiphp: Slot [22] registered [ 0.326071] acpiphp: Slot [23] registered [ 0.327092] acpiphp: Slot [24] registered [ 0.329081] acpiphp: Slot [25] registered [ 0.330071] acpiphp: Slot [26] registered [ 0.332071] acpiphp: Slot [27] registered [ 0.333078] acpiphp: Slot [28] registered [ 0.335074] acpiphp: Slot [29] registered [ 0.336155] acpiphp: Slot [30] registered [ 0.337076] acpiphp: Slot [31] registered [ 0.339042] PCI host bridge to bus 0000:00 [ 0.340012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.345020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.348019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.361009] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365535] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374792] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380464] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.385012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390633] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393813] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.396055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399589] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.409014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.433012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.439893] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.467021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.489015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.501314] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.508015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.543920] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.551013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.558015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.575015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.585959] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.594013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.600015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.617017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626643] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.640014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667392] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.674015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.680015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.695014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705801] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711335] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713289] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.716233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720172] iommu: Default domain type: Passthrough [ 0.721000] SCSI subsystem initialized [ 0.721000] ACPI: bus type USB registered [ 0.721000] usbcore: registered new interface driver usbfs [ 0.722087] usbcore: registered new interface driver hub [ 0.724068] usbcore: registered new device driver usb [ 0.725178] pps_core: LinuxPPS API ver. 1 registered [ 0.727024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.729052] PTP clock support registered [ 0.731207] EDAC MC: Ver: 3.0.0 [ 0.733209] PCI: Using ACPI for IRQ routing [ 0.734800] NetLabel: Initializing [ 0.736086] NetLabel: domain hash size = 128 [ 0.737013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738093] NetLabel: unlabeled traffic allowed by default [ 0.740102] vgaarb: loaded [ 0.741236] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748817] clocksource: Switched to clocksource kvm-clock [ 0.859875] VFS: Disk quotas dquot_6.6.0 [ 0.860941] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.862612] *** VALIDATE ramfs *** [ 0.863429] *** VALIDATE hugetlbfs *** [ 0.864480] pnp: PnP ACPI init [ 0.866258] pnp: PnP ACPI: found 6 devices [ 0.892157] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894480] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.896360] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898111] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899878] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.901505] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.903467] NET: Registered protocol family 2 [ 0.905356] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.909259] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.911936] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.916327] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.919075] TCP: Hash tables configured (established 65536 bind 65536) [ 0.921330] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.923354] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.925264] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927234] NET: Registered protocol family 1 [ 0.929251] RPC: Registered named UNIX socket transport module. [ 0.931166] RPC: Registered udp transport module. [ 0.932424] RPC: Registered tcp transport module. [ 0.933786] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.935716] NET: Registered protocol family 44 [ 0.936898] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.938475] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.940093] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.941983] PCI: CLS 0 bytes, default 64 [ 0.943363] Unpacking initramfs... [ 2.274766] debug: unmapping init [mem 0xffff94817cc54000-0xffff94817ffbffff] [ 2.278354] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.279866] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.281814] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.803267] Initialise system trusted keyrings [ 2.804773] Key type blacklist registered [ 2.806310] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813923] zbud: loaded [ 2.817341] *** VALIDATE nfs *** [ 2.818424] *** VALIDATE nfs4 *** [ 2.819692] pstore: using deflate compression [ 2.822531] Platform Keyring initialized [ 2.921157] NET: Registered protocol family 38 [ 2.922584] Key type asymmetric registered [ 2.923930] Asymmetric key parser 'x509' registered [ 2.925431] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.927840] io scheduler mq-deadline registered [ 2.929331] io scheduler kyber registered [ 2.930731] io scheduler bfq registered [ 2.932550] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.935106] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.936990] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.938786] ACPI: Power Button [PWRF] [ 2.943656] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.950128] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.963135] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.969839] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.987743] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.015953] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.044040] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.050337] Non-volatile memory driver v1.3 [ 3.051749] Linux agpgart interface v0.103 [ 3.079413] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.081769] vda: detected capacity change from 0 to 74702848 [ 3.095936] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.098428] vdb: detected capacity change from 0 to 1073741824 [ 3.113701] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116552] vdc: detected capacity change from 0 to 2621440000 [ 3.129383] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.131684] vdd: detected capacity change from 0 to 2621440000 [ 3.144504] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.147532] vde: detected capacity change from 0 to 4294967296 [ 3.161955] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.165640] vdf: detected capacity change from 0 to 4294967296 [ 3.178677] libphy: Fixed MDIO Bus: probed [ 3.187299] usbcore: registered new interface driver usbserial_generic [ 3.189797] usbserial: USB Serial support registered for generic [ 3.191994] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.196304] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.198240] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.200691] mousedev: PS/2 mouse device common for all mice [ 3.203542] rtc_cmos 00:05: RTC can wake from S4 [ 3.206500] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.208209] rtc_cmos 00:05: registered as rtc0 [ 3.211293] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.213666] intel_pstate: CPU model not supported [ 3.216264] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.218535] hid: raw HID events driver (C) Jiri Kosina [ 3.221869] usbcore: registered new interface driver usbhid [ 3.223798] usbhid: USB HID core driver [ 3.223990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.225244] drop_monitor: Initializing network drop monitor service [ 3.230457] Initializing XFRM netlink socket [ 3.232322] NET: Registered protocol family 10 [ 3.235124] Segment Routing with IPv6 [ 3.236257] NET: Registered protocol family 17 [ 3.238098] mpls_gso: MPLS GSO support [ 3.242770] RAS: Correctable Errors collector initialized. [ 3.244854] AVX version of gcm_enc/dec engaged. [ 3.246337] AES CTR mode by8 optimization enabled [ 3.326686] sched_clock: Marking stable (3326656600, 0)->(4185015108, -858358508) [ 3.330823] registered taskstats version 1 [ 3.332261] Loading compiled-in X.509 certificates [ 3.334819] zswap: loaded using pool lzo/zbud [ 3.359843] Key type big_key registered [ 3.372915] Key type encrypted registered [ 3.374760] ima: No TPM chip found, activating TPM-bypass! [ 3.377342] ima: Allocated hash algorithm: sha1 [ 3.379205] ima: No architecture policies found [ 3.381270] evm: Initialising EVM extended attributes: [ 3.383383] evm: security.selinux [ 3.384784] evm: security.ima [ 3.385904] evm: security.capability [ 3.387453] evm: HMAC attrs: 0x1 [ 3.390290] rtc_cmos 00:05: setting system clock to 2026-08-04 20:44:00 UTC (1785876240) [ 3.397437] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.401162] debug: unmapping init [mem 0xffffffffa2382000-0xffffffffa2658fff] [ 3.408171] Write protecting the kernel read-only data: 28672k [ 3.410838] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 3.413586] debug: unmapping init [mem 0xffffffffa1314000-0xffffffffa13fffff] [ 3.451573] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.460458] systemd[1]: Detected virtualization kvm. [ 3.462735] systemd[1]: Detected architecture x86-64. [ 3.465256] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.487609] systemd[1]: No hostname configured. [ 3.489644] systemd[1]: Set hostname to . [ 3.491564] random: systemd: uninitialized urandom read (16 bytes read) [ 3.494337] systemd[1]: Initializing machine ID from random generator. [ 3.632841] random: systemd: uninitialized urandom read (16 bytes read) [ 3.636461] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.640840] random: systemd: uninitialized urandom read (16 bytes read) [ 3.647638] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.652224] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.323956] device-mapper: uevent: version 1.0.3 [ 4.325892] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.089996] virtio_net virtio0 ens2: renamed from eth0 [ 5.148502] random: fast init done [ 5.234247] scsi host0: ata_piix [ 5.271992] scsi host1: ata_piix [ 5.277337] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.279298] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.965872] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.892668] random: crng init done [ 9.894231] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.379231] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.572293] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.856133] SELinux: Disabled at runtime. [ 11.917871] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.925496] systemd[1]: Detected virtualization kvm. [ 11.927346] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.447212] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.450612] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.455588] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.461278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.464345] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.473052] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.477581] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.692899] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.042765] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.378124] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.395248] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.528232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.537723] EDAC sbridge: Ver: 1.1.2 [ 14.996932] Key type dns_resolver registered [ 15.289653] NFS: Registering the id_resolver key type [ 15.291933] Key type id_resolver registered [ 15.293380] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 54.903911] libcfs: loading out-of-tree module taints kernel. [ 54.979208] Key type ._llcrypt registered [ 54.984982] Key type .llcrypt registered [ 55.166788] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 84.424119] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 86.899600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.934996] alg: No test for adler32 (adler32-zlib) [ 89.024203] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 90.908223] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 92.712262] Key type lgssc registered [ 96.401298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.600078] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 124.619898] hrtimer: interrupt took 4274102 ns [ 180.995915] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 196.188198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.254201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.658725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 197.730676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 197.936598] Lustre: lustre-MDT0000: new disk, initializing [ 198.105113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.150692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.347988] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.156933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.352415] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 224.418563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 224.427021] Lustre: Skipped 1 previous similar message [ 224.582920] Lustre: lustre-MDT0001: new disk, initializing [ 224.692698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.750528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 224.772446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.080724] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.258068] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 256.770036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.302715] Lustre: lustre-OST0000: new disk, initializing [ 257.309296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 257.325528] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.413489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.303272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.321727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.529218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 267.235950] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.321843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.617485] Lustre: lustre-OST0001: new disk, initializing [ 287.619672] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 287.629543] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.850936] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 297.036387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 297.063391] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.187476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 298.556965] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 316.823309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 324.709658] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 332.751684] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 338.444478] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 345.277865] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 348.884225] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 352.593579] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 354.575864] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 4 16:49:51 EDT 2026 [ 377.319142] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 379.198833] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 398.817788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 398.823706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.851798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.850310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.871390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.875424] Lustre: Skipped 1 previous similar message [ 399.885882] Lustre: Skipped 3 previous similar messages [ 403.934455] Lustre: server umount lustre-MDT0000 complete [ 408.973895] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785876646 with bad export cookie 12498867953519528745 [ 408.977367] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 408.989037] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 410.080732] LustreError: 8001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.081148] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.091592] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.098435] LustreError: 8001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 410.138539] Lustre: Skipped 2 previous similar messages [ 413.098151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 413.100438] Lustre: Skipped 1 previous similar message [ 413.105788] LustreError: 8001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.137961] LustreError: 8001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 418.279382] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 418.286096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 418.314404] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 418.340988] Lustre: Skipped 1 previous similar message [ 423.392306] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 423.767875] Lustre: server umount lustre-MDT0001 complete [ 429.349694] Lustre: server umount lustre-OST0000 complete [ 433.844686] Lustre: server umount lustre-OST0001 complete [ 443.272494] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 459.405649] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 508.747724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.219695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 509.299235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 509.450655] Lustre: lustre-MDT0000: new disk, initializing [ 509.650612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.697221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 516.609209] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 532.883596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.015525] Lustre: 15112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 533.085502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 533.090180] Lustre: Skipped 1 previous similar message [ 533.208503] Lustre: lustre-MDT0001: new disk, initializing [ 533.414914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 533.469173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 533.487423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 541.507463] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 557.945855] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 565.580107] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.490117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 577.149891] Lustre: lustre-OST0000: new disk, initializing [ 577.165741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 577.187892] Lustre: 17085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 577.374411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.209061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 579.242580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 579.497711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 587.202080] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 600.012260] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 608.049549] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 608.268956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 613.730982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 614.139667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 620.519474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 620.532741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.562964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 620.572539] Lustre: Skipped 3 previous similar messages [ 622.885641] Lustre: server umount lustre-OST0000 complete [ 630.756120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.779088] Lustre: Skipped 2 previous similar messages [ 634.618861] Lustre: server umount lustre-MDT0000 complete [ 635.874209] LustreError: 15124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.907142] LustreError: 15124:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 640.036320] LustreError: 15118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.067342] LustreError: 15118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 641.357206] LustreError: 15103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785876878 with bad export cookie 12498867953519530278 [ 641.358084] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.381649] LustreError: 15103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 641.864356] Lustre: server umount lustre-MDT0001 complete [ 647.278355] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 16:54:43 (1785876883) [ 657.855419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.624514] LustreError: 19351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.707706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.634222] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 676.074335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.765746] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 692.067149] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 699.804342] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 709.832411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.170879] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 730.720873] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 742.369238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 742.380212] LustreError: Skipped 1 previous similar message [ 742.384975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.411309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 742.413565] Lustre: Skipped 2 previous similar messages [ 744.426686] LustreError: 21846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 744.433210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 744.446497] LustreError: 21846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 744.773902] Lustre: server umount lustre-OST0000 complete [ 753.637589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 757.084849] Lustre: server umount lustre-MDT0000 complete [ 762.460403] LustreError: 19332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785876999 with bad export cookie 12498867953519531160 [ 762.470124] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.473218] LustreError: 19332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 762.981425] Lustre: server umount lustre-MDT0001 complete [ 777.304311] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 781.453471] Key type lgssc unregistered [ 781.877790] LNet: 22894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.881531] LNetError: 22894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.915335] LNet: Removed LNI 192.168.201.121@tcp [ 783.298243] Key type .llcrypt unregistered [ 783.303578] Key type ._llcrypt unregistered [ 815.093920] Key type ._llcrypt registered [ 815.095696] Key type .llcrypt registered [ 815.285398] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 816.560576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.609173] alg: No test for adler32 (adler32-zlib) [ 818.140264] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 818.546853] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 820.385607] Key type lgssc registered [ 821.919355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 830.004753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 830.026686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.583854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 837.126406] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 841.698202] LustreError: 24321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 841.716623] LustreError: 24321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 846.826886] LustreError: 24322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.722364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.367996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.100856] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 866.232625] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 873.224894] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 881.930422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.550627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 889.526676] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 900.886610] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 901.032871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 924.644697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.656658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.680165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.492573] Lustre: server umount lustre-OST0000 complete [ 931.810627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.835676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.929630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.932073] Lustre: Skipped 2 previous similar messages [ 941.027072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.035799] Lustre: Skipped 1 previous similar message [ 941.212778] Lustre: server umount lustre-MDT0000 complete [ 944.951939] LustreError: 24303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785877182 with bad export cookie 7506255249864451952 [ 944.973541] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 945.612270] Lustre: server umount lustre-MDT0001 complete [ 961.110645] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 965.276565] Key type lgssc unregistered [ 965.661954] LNet: 28307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.668376] LNetError: 28307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.695484] LNet: Removed LNI 192.168.201.121@tcp [ 967.008075] Key type .llcrypt unregistered [ 967.009707] Key type ._llcrypt unregistered [ 998.813197] Key type ._llcrypt registered [ 998.817174] Key type .llcrypt registered [ 999.019928] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1002.375802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.456097] alg: No test for adler32 (adler32-zlib) [ 1004.203718] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 1004.661282] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1006.400159] Key type lgssc registered [ 1008.012360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1016.409909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.452419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.216759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.881253] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1028.586310] LustreError: 29727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.614275] LustreError: 29727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1033.700945] LustreError: 29728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.906290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.371286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1040.829324] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1050.867762] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1058.090459] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1075.266749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1075.819698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.924641] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1088.150144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1094.003525] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.851208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.851722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.876307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1108.974516] LustreError: 31906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.006495] LustreError: 31906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1109.551682] Lustre: server umount lustre-OST0000 complete [ 1117.153170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1117.156662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.198353] Lustre: Skipped 1 previous similar message [ 1117.220082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.232697] Lustre: Skipped 1 previous similar message [ 1119.202820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.236164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.033785] Lustre: server umount lustre-MDT0000 complete [ 1124.344060] LustreError: 29728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.967124] LustreError: 29709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785877366 with bad export cookie 15961255776467846875 [ 1128.984136] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.984815] LustreError: 29709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1129.634695] Lustre: server umount lustre-MDT0001 complete [ 1143.141934] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1147.363735] Key type lgssc unregistered [ 1147.577767] LNet: 33628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.581419] LNetError: 33628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.650601] LNet: Removed LNI 192.168.201.121@tcp [ 1149.755186] Key type .llcrypt unregistered [ 1149.758204] Key type ._llcrypt unregistered [ 1180.197150] Key type ._llcrypt registered [ 1180.199535] Key type .llcrypt registered [ 1180.327871] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1181.628928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.720012] alg: No test for adler32 (adler32-zlib) [ 1183.222373] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 1183.707243] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1185.520156] Key type lgssc registered [ 1187.135723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1197.202225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1197.238555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.847778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.747478] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1208.295841] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.317797] LustreError: 35056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1213.417837] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.378307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.149776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1225.540811] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1237.027996] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1244.950848] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1254.577389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.313229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.165870] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1269.619969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1275.406492] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1289.131027] Lustre: server umount lustre-OST0000 complete [ 1294.818090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.831057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.846391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.334236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.359815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.363379] Lustre: Skipped 1 previous similar message [ 1300.452113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1300.464579] Lustre: Skipped 1 previous similar message [ 1304.662491] Lustre: server umount lustre-MDT0000 complete [ 1305.504183] Lustre: 34368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785877526/real 1785877526] req@00000000659cbf9d x1872628201587328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785877542 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1305.550930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.576826] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1305.596509] LustreError: 35055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1306.592285] Lustre: 34367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785877527/real 1785877527] req@000000009b512168 x1872628201588352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785877543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1310.479963] LustreError: 36903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785877547 with bad export cookie 8203850312771805152 [ 1310.491963] LustreError: 36903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1310.500995] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.224074] Lustre: server umount lustre-MDT0001 complete [ 1325.029692] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1329.648639] Key type lgssc unregistered [ 1330.246637] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.271415] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.337176] LNet: Removed LNI 192.168.201.121@tcp [ 1333.321048] Key type .llcrypt unregistered [ 1333.322821] Key type ._llcrypt unregistered [ 1369.329586] Key type ._llcrypt registered [ 1369.331328] Key type .llcrypt registered [ 1369.538368] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1370.984482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.119499] alg: No test for adler32 (adler32-zlib) [ 1372.398627] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 1372.818308] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1374.684153] Key type lgssc registered [ 1376.451479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.090266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.142208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.931235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1396.210051] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1399.268249] LustreError: 40018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.285625] LustreError: 40018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1404.390062] LustreError: 40019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.294670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.762490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.027683] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1422.777753] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.540877] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.966427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.569302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.364312] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1459.068290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1467.858134] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1478.628726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1478.643102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.681187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1479.654123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.667474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.312631] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1481.327064] Lustre: Skipped 1 previous similar message [ 1483.496366] Lustre: server umount lustre-OST0000 complete [ 1509.346233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.359356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1509.374422] Lustre: Skipped 1 previous similar message [ 1509.423936] Lustre: Skipped 1 previous similar message [ 1513.868985] Lustre: server umount lustre-MDT0000 complete [ 1514.464930] LustreError: 41135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.484092] LustreError: 41135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1518.817994] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785877755 with bad export cookie 16867567227838669209 [ 1518.840248] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1518.846374] LustreError: 39999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1533.408135] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1533.885958] Lustre: server umount lustre-MDT0001 complete [ 1554.667710] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1558.480423] Key type lgssc unregistered [ 1558.904677] LNet: 43756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.915556] LNetError: 43756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.966868] LNet: Removed LNI 192.168.201.121@tcp [ 1559.972871] Key type .llcrypt unregistered [ 1559.975966] Key type ._llcrypt unregistered [ 1594.894795] Key type ._llcrypt registered [ 1594.896389] Key type .llcrypt registered [ 1595.115285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1597.720892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.790326] alg: No test for adler32 (adler32-zlib) [ 1599.104105] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 1599.699247] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1601.464199] Key type lgssc registered [ 1603.642183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1613.262154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1613.295968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.939278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.307609] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1625.065635] LustreError: 45171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.103739] LustreError: 45171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1630.180325] LustreError: 45172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.285206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1634.983260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1641.661862] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1652.916979] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1660.452542] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1669.598637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1670.291749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.891076] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1685.615090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1689.586522] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1697.882040] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1701.346710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1701.357225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.385324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1701.390201] Lustre: Skipped 1 previous similar message [ 1702.966412] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 1704.652964] Lustre: server umount lustre-MDT0000 complete [ 1706.472517] LustreError: 45901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.508352] LustreError: 45901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1709.366577] LustreError: 45151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785877946 with bad export cookie 17547669563396511965 [ 1709.387869] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.599646] LustreError: 45166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.609763] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.628978] LustreError: 45166:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1711.672810] Lustre: Skipped 2 previous similar messages [ 1711.684947] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1716.272918] Lustre: server umount lustre-MDT0001 complete [ 1732.650609] Lustre: server umount lustre-OST0000 complete [ 1758.748446] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1763.712809] Key type lgssc unregistered [ 1764.461707] LNet: 49040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.466119] LNetError: 49040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.523706] LNet: Removed LNI 192.168.201.121@tcp [ 1766.135140] Key type .llcrypt unregistered [ 1766.139457] Key type ._llcrypt unregistered [ 1809.226306] Key type ._llcrypt registered [ 1809.230023] Key type .llcrypt registered [ 1809.398679] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1812.726406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.788385] alg: No test for adler32 (adler32-zlib) [ 1814.537798] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 1815.258693] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1817.048269] Key type lgssc registered [ 1819.236585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1831.974169] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1831.986617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1847.520388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1852.640898] LustreError: 50431:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 1879.328205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1886.428954] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1897.021798] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2066.899447] Lustre: server umount lustre-OST0000 complete [ 2094.312151] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2098.152614] Key type lgssc unregistered [ 2098.674347] LNet: 52113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.690753] LNetError: 52113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.766689] LNet: Removed LNI 192.168.201.121@tcp [ 2100.022919] Key type .llcrypt unregistered [ 2100.024785] Key type ._llcrypt unregistered [ 2132.820432] Key type ._llcrypt registered [ 2132.828161] Key type .llcrypt registered [ 2132.976450] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2135.525852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.563646] alg: No test for adler32 (adler32-zlib) [ 2136.926206] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 2137.363299] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2139.152277] Key type lgssc registered [ 2141.026261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2152.258556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2152.284310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.050677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.348784] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2164.204317] LustreError: 53531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.232409] LustreError: 53531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2169.322076] LustreError: 53532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.759537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.385715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2179.060134] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2189.762698] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2199.939445] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2209.783279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.146856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.724460] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2224.591145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2231.431862] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2234.849899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2234.861989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.877200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.887959] Lustre: Skipped 1 previous similar message [ 2239.970687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2239.988424] Lustre: Skipped 2 previous similar messages [ 2241.027560] Lustre: server umount lustre-OST0000 complete [ 2250.212543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.215401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2250.233943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.980274] Lustre: server umount lustre-MDT0000 complete [ 2255.342340] LustreError: 53532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2255.381446] LustreError: 53532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2259.502268] LustreError: 53511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785878496 with bad export cookie 4086033336063815830 [ 2259.510095] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.523953] LustreError: 53511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2260.123202] Lustre: server umount lustre-MDT0001 complete [ 2280.758375] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2286.191249] Key type lgssc unregistered [ 2286.730334] LNet: 57093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.758094] LNetError: 57093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.803308] LNet: Removed LNI 192.168.201.121@tcp [ 2289.181682] Key type .llcrypt unregistered [ 2289.183438] Key type ._llcrypt unregistered [ 2338.561708] Key type ._llcrypt registered [ 2338.565454] Key type .llcrypt registered [ 2338.768775] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2340.833590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2341.174110] alg: No test for adler32 (adler32-zlib) [ 2342.603749] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 2342.953960] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2344.834296] Key type lgssc registered [ 2346.423513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2360.238301] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2360.263536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2375.968868] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2381.090195] LustreError: 58483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 2407.712416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.931772] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2430.926652] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2439.889841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.787778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.919542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2447.461223] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2448.357228] LustreError: 59521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.380629] LustreError: 59521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2453.481465] LustreError: 58508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.503801] LustreError: 58508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2458.595873] LustreError: 59615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.631322] LustreError: 59615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2459.997105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.598618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2465.535992] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2475.719720] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2482.345162] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2486.246542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2486.259281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2486.262503] Lustre: Skipped 1 previous similar message [ 2486.265250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2486.287270] Lustre: Skipped 1 previous similar message [ 2491.309968] Lustre: server umount lustre-OST0000 complete [ 2516.962042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2516.964075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.977681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2517.005106] Lustre: Skipped 1 previous similar message [ 2522.081439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.090939] Lustre: Skipped 2 previous similar messages [ 2522.643544] Lustre: server umount lustre-MDT0000 complete [ 2526.177868] LustreError: 59520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.185382] LustreError: 59520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2527.528086] LustreError: 58489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785878764 with bad export cookie 18147803687702805678 [ 2527.541207] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.549867] LustreError: 58489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2534.162048] Lustre: server umount lustre-MDT0001 complete [ 2553.063825] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2556.975459] Key type lgssc unregistered [ 2557.345681] LNet: 62253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.350862] LNetError: 62253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.369283] LNet: Removed LNI 192.168.201.121@tcp [ 2558.522890] Key type .llcrypt unregistered [ 2558.526339] Key type ._llcrypt unregistered [ 2584.257055] Key type ._llcrypt registered [ 2584.262100] Key type .llcrypt registered [ 2584.483874] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2586.268867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2586.303418] alg: No test for adler32 (adler32-zlib) [ 2587.453454] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 2587.849350] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2589.656136] Key type lgssc registered [ 2591.277818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.505627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.526163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.182479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.804920] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2613.734277] LustreError: 63683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.784326] LustreError: 63683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2618.859139] LustreError: 63684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.914404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.406261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2628.246713] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2637.836487] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.187256] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2653.728568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2654.361366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.187072] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2673.857174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2675.331652] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2694.653777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.656074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.666729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.724308] Lustre: Skipped 1 previous similar message [ 2699.766809] LustreError: 66125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.808468] LustreError: 66125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2700.332325] Lustre: server umount lustre-OST0000 complete [ 2708.452122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2708.468606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.495269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2708.510026] Lustre: Skipped 1 previous similar message [ 2710.000719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2710.021245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.954707] Lustre: server umount lustre-MDT0000 complete [ 2715.107308] LustreError: 63683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.142086] LustreError: 63683:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2718.656180] LustreError: 63663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785878955 with bad export cookie 5656283209362599274 [ 2718.660990] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.687937] LustreError: 63663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2719.462130] Lustre: server umount lustre-MDT0001 complete [ 2735.031647] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 2739.179840] Key type lgssc unregistered [ 2739.493868] LNet: 67241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2739.502225] LNetError: 67241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2739.531156] LNet: Removed LNI 192.168.201.121@tcp [ 2740.548241] Key type .llcrypt unregistered [ 2740.554301] Key type ._llcrypt unregistered [ 2781.058275] Key type ._llcrypt registered [ 2781.066268] Key type .llcrypt registered [ 2781.278877] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2783.498579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2783.862755] alg: No test for adler32 (adler32-zlib) [ 2785.217895] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 2785.633107] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 2787.488207] Key type lgssc registered [ 2789.674996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.578510] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 17:30:42 (1785879042) [ 2815.934597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2815.979782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.515247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.500669] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2827.746667] LustreError: 68950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.762875] LustreError: 68950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2832.873529] LustreError: 68951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.952098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.742594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2843.298719] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2854.493736] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2864.779055] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2874.559400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.384570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.742459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2888.706294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2897.851581] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.027941] Lustre: Failing over lustre-MDT0000 [ 2908.129479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.144235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.175670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.624874] Lustre: server umount lustre-MDT0000 complete [ 2909.153681] LustreError: 68951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.156691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.194255] LustreError: 68951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2909.207437] Lustre: Skipped 1 previous similar message [ 2911.837457] LustreError: 70049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.926091] LustreError: 68945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.939023] LustreError: 68945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2921.934990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.219535] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.394992] Lustre: *** cfs_fail_loc=135, val=0*** [ 2922.541604] LustreError: 72044:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2922.558332] LustreError: 72044:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -2 [ 2922.579558] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2922.579558] [ 2922.593663] LustreError: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2922.616974] LustreError: 72033:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2922.637428] LustreError: 72033:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2922.650601] LustreError: 72033:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2922.668981] LustreError: 72033:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2922.734915] Lustre: server umount lustre-MDT0000 complete [ 2922.741612] LustreError: 72033:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2925.055605] LustreError: 68946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.077455] LustreError: 68946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2933.310594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.451853] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.472031] LustreError: 68942:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2933.966966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.094189] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2938.404283] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2939.418977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2939.499326] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2939.557040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2940.497548] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2953.185571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2953.206728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.245199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.725466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2954.750290] Lustre: Skipped 1 previous similar message [ 2959.848044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.862018] Lustre: Skipped 1 previous similar message [ 2964.962203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2964.978903] Lustre: Skipped 1 previous similar message [ 2965.984136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2966.664116] Lustre: server umount lustre-OST0000 complete [ 2975.214134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.222862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.225360] Lustre: Skipped 2 previous similar messages [ 2975.255746] Lustre: Skipped 1 previous similar message [ 2978.822469] Lustre: server umount lustre-MDT0000 complete [ 2980.320996] LustreError: 69878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.353870] LustreError: 69878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2986.352557] LustreError: 68931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785879223 with bad export cookie 15576466295025433580 [ 2986.374629] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.397949] LustreError: 68931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2987.477744] Lustre: server umount lustre-MDT0001 complete [ 3008.365931] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3011.628293] Key type lgssc unregistered [ 3011.927689] LNet: 73849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3011.939855] LNetError: 73849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3011.952837] LNet: Removed LNI 192.168.201.121@tcp [ 3012.992558] Key type .llcrypt unregistered [ 3012.998581] Key type ._llcrypt unregistered [ 3045.090909] Key type ._llcrypt registered [ 3045.096719] Key type .llcrypt registered [ 3045.281910] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3046.888655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3047.184796] alg: No test for adler32 (adler32-zlib) [ 3048.428587] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 3048.925150] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3050.688158] Key type lgssc registered [ 3052.135355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3062.273366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3062.307202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.681974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.922550] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3073.003785] LustreError: 75268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3073.035398] LustreError: 75268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3078.132721] LustreError: 75269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.771346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.154230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.310214] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3098.874976] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3106.239231] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3115.196070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.726641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3124.673378] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3133.177049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3134.533533] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.847093] Lustre: Failing over lustre-MDT0000 [ 3141.733547] LustreError: 75264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.627620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3142.640447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.125631] Lustre: server umount lustre-MDT0000 complete [ 3143.668711] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.691916] Lustre: Skipped 1 previous similar message [ 3146.808653] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.849103] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3151.933304] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.956774] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3156.765666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.033812] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.303324] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3157.610744] LustreError: 78359:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3157.623361] LustreError: 78359:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -9 [ 3157.643622] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3157.643622] [ 3157.670322] LustreError: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3157.691833] LustreError: 78347:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3157.706356] LustreError: 78347:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3157.728345] LustreError: 78347:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3157.748285] LustreError: 78347:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3157.828578] Lustre: server umount lustre-MDT0000 complete [ 3157.830536] LustreError: 78347:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3162.168916] LustreError: 75265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.218525] LustreError: 75265:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 3168.441752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.550186] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.565489] LustreError: 75260:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3168.833486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.905648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3172.416673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3173.902531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3173.948336] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3173.982678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3175.024331] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3187.682077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3187.704935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.729332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.226295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.239345] Lustre: Skipped 1 previous similar message [ 3191.361330] Lustre: server umount lustre-OST0000 complete [ 3199.470558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.474709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.482486] Lustre: Skipped 1 previous similar message [ 3199.486434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.578049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.593883] Lustre: Skipped 2 previous similar messages [ 3209.699570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3209.707327] Lustre: Skipped 1 previous similar message [ 3211.232192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3211.589385] Lustre: server umount lustre-MDT0000 complete [ 3214.818906] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3214.840882] LustreError: 76376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3216.315041] LustreError: 75248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785879453 with bad export cookie 4885714792208285711 [ 3216.324365] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.331066] LustreError: 75248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3217.154284] Lustre: server umount lustre-MDT0001 complete [ 3235.801430] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3242.341967] Key type lgssc unregistered [ 3242.884493] LNet: 80167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3242.896323] LNetError: 80167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3242.934382] LNet: Removed LNI 192.168.201.121@tcp [ 3244.372684] Key type .llcrypt unregistered [ 3244.376621] Key type ._llcrypt unregistered [ 3262.657755] Key type ._llcrypt registered [ 3262.664236] Key type .llcrypt registered [ 3262.828564] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3263.377582] Key type .llcrypt unregistered [ 3263.379141] Key type ._llcrypt unregistered [ 3300.505789] Key type ._llcrypt registered [ 3300.511611] Key type .llcrypt registered [ 3300.723894] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3302.408053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3302.526952] alg: No test for adler32 (adler32-zlib) [ 3303.970855] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 3304.414701] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3306.057991] Key type lgssc registered [ 3308.299238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.518525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.530366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.002344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.540540] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3333.611219] LustreError: 82078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.641631] LustreError: 82078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3338.739219] LustreError: 82079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.849393] LustreError: 82078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3344.529618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.171620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.923319] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3361.131774] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3369.044276] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3379.342645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3379.959780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.146542] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3393.278733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 3405.618894] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3417.058227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3417.073615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3417.096954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.111110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.112046] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.121720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.407771] Lustre: server umount lustre-OST0000 complete [ 3427.303267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3427.317759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.354945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.361551] Lustre: Skipped 1 previous similar message [ 3429.347743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.369930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.378600] Lustre: Skipped 1 previous similar message [ 3432.672311] Lustre: server umount lustre-MDT0000 complete [ 3434.467703] LustreError: 84076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3434.489208] LustreError: 84076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3437.055681] LustreError: 82059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785879674 with bad export cookie 11273024993538591438 [ 3437.084949] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.682692] Lustre: server umount lustre-MDT0001 complete [ 3453.935414] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3458.198443] Key type lgssc unregistered [ 3458.642423] LNet: 85620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3458.647493] LNetError: 85620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3458.674122] LNet: Removed LNI 192.168.201.121@tcp [ 3459.979164] Key type .llcrypt unregistered [ 3459.981577] Key type ._llcrypt unregistered [ 3497.263090] Key type ._llcrypt registered [ 3497.269235] Key type .llcrypt registered [ 3497.400627] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3498.634318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3498.844157] alg: No test for adler32 (adler32-zlib) [ 3499.843889] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 3500.024236] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3501.640206] Key type lgssc registered [ 3503.459576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3512.345978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3512.368288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.736559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.125044] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3523.055207] LustreError: 87034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3523.079497] LustreError: 87034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3528.162404] LustreError: 87035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.476296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.107882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3538.658765] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3551.200720] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3560.578641] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3571.170649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.551731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.458372] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3583.883658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3593.067649] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3603.427276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.435955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.447577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3604.453409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3604.464053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3604.465823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3604.493678] Lustre: Skipped 1 previous similar message [ 3608.675132] Lustre: server umount lustre-OST0000 complete [ 3619.298311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3619.319092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3619.341645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3621.678964] Lustre: server umount lustre-MDT0000 complete [ 3624.944314] LustreError: 87763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3624.978617] LustreError: 87763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3628.441571] LustreError: 87015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785879865 with bad export cookie 5355206275464941728 [ 3628.452185] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3628.452594] LustreError: 87015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3629.062580] Lustre: server umount lustre-MDT0001 complete [ 3644.781997] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3648.424680] Key type lgssc unregistered [ 3648.721685] LNet: 90573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3648.726677] LNetError: 90573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3648.742800] LNet: Removed LNI 192.168.201.121@tcp [ 3649.931695] Key type .llcrypt unregistered [ 3649.936216] Key type ._llcrypt unregistered [ 3684.579414] Key type ._llcrypt registered [ 3684.583899] Key type .llcrypt registered [ 3684.792453] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3687.232531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3687.272514] alg: No test for adler32 (adler32-zlib) [ 3688.631907] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 3689.105825] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3690.920717] Key type lgssc registered [ 3692.103748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3702.210955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3702.239947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.932328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.159432] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3714.537857] LustreError: 91990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3714.568141] LustreError: 91990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3719.651806] LustreError: 91991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3723.187432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3723.730630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3728.544301] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3736.701395] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3743.347807] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3752.678716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3753.331097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.407554] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3775.219367] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.893375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3796.457129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3796.459165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.473523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3796.503446] Lustre: Skipped 1 previous similar message [ 3796.519721] Lustre: Skipped 1 previous similar message [ 3801.578614] LustreError: 94483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3801.602898] LustreError: 94483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3801.779943] Lustre: server umount lustre-OST0000 complete [ 3810.785524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3810.809410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.868501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.815840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.829169] Lustre: server umount lustre-MDT0000 complete [ 3816.929822] LustreError: 91991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.967586] LustreError: 91991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3821.809022] LustreError: 91972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785880058 with bad export cookie 11233090931750759705 [ 3821.817976] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3821.829243] LustreError: 91972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3823.105199] Lustre: server umount lustre-MDT0001 complete [ 3842.970203] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3847.959262] Key type lgssc unregistered [ 3848.257721] LNet: 95575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3848.265267] LNetError: 95575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3848.291331] LNet: Removed LNI 192.168.201.121@tcp [ 3849.207638] Key type .llcrypt unregistered [ 3849.209628] Key type ._llcrypt unregistered [ 3885.577753] Key type ._llcrypt registered [ 3885.580353] Key type .llcrypt registered [ 3885.720868] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3888.520778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3888.613906] alg: No test for adler32 (adler32-zlib) [ 3890.114986] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 3890.596634] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3892.435489] Key type lgssc registered [ 3894.930586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3908.681519] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3908.737284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.384624] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3929.504335] LustreError: 96976:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 3956.194096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.730502] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3975.485814] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3983.152565] Lustre: server umount lustre-OST0000 complete [ 3995.148300] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 17:50:31 (1785880231) [ 4003.692051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.018351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.440397] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.461034] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4009.978457] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4014.569432] LustreError: 98618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.683753] LustreError: 98617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.482638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.793019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.308933] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4034.606383] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4041.854743] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4051.025214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.400483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.132345] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4067.887302] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.987088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 4089.314074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.321575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.331418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4089.351126] Lustre: Skipped 1 previous similar message [ 4094.216538] Lustre: server umount lustre-OST0000 complete [ 4099.556337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.575821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.578240] Lustre: Skipped 1 previous similar message [ 4104.673635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4104.681350] Lustre: Skipped 1 previous similar message [ 4109.793281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4109.796820] Lustre: Skipped 1 previous similar message [ 4113.378635] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4113.776424] Lustre: server umount lustre-MDT0000 complete [ 4114.916994] LustreError: 98614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.951713] LustreError: 98614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4120.045219] LustreError: 99345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.100493] LustreError: 99345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4120.758873] LustreError: 98598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785880357 with bad export cookie 6199850101021871202 [ 4120.798427] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.500451] Lustre: server umount lustre-MDT0001 complete [ 4146.326679] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4149.990226] Key type lgssc unregistered [ 4150.477722] LNet: 102355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4150.484705] LNetError: 102355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4150.500416] LNet: Removed LNI 192.168.201.121@tcp [ 4151.214707] Key type .llcrypt unregistered [ 4151.216453] Key type ._llcrypt unregistered [ 4200.657279] Key type ._llcrypt registered [ 4200.660524] Key type .llcrypt registered [ 4200.732079] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4201.721868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4201.851985] alg: No test for adler32 (adler32-zlib) [ 4203.036919] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 4203.357188] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4204.977458] Key type lgssc registered [ 4206.849972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4216.130897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4216.149434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.532488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.190075] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4227.101554] LustreError: 104543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.160783] LustreError: 104543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4232.172453] LustreError: 104544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4234.311831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4234.800892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4240.920884] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4249.757303] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4257.665579] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4268.567060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4268.928242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.446843] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4287.343695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 4287.548931] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4302.821612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4302.823670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4302.839390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.940070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4307.945995] Lustre: Skipped 2 previous similar messages [ 4308.619831] Lustre: server umount lustre-OST0000 complete [ 4313.061684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.075371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.083428] Lustre: Skipped 2 previous similar messages [ 4318.177980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.193158] Lustre: Skipped 2 previous similar messages [ 4323.308531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4327.400481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4327.617931] Lustre: server umount lustre-MDT0000 complete [ 4328.418506] LustreError: 104544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4328.440616] LustreError: 104544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4332.243476] LustreError: 104525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785880569 with bad export cookie 8746470844810889538 [ 4332.260025] LustreError: 104525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4332.261265] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4333.078508] Lustre: server umount lustre-MDT0001 complete [ 4346.634780] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4349.764255] Key type lgssc unregistered [ 4350.151232] LNet: 108105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4350.157179] LNetError: 108105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4350.185110] LNet: Removed LNI 192.168.201.121@tcp [ 4351.381347] Key type .llcrypt unregistered [ 4351.387674] Key type ._llcrypt unregistered [ 4377.060762] Key type ._llcrypt registered [ 4377.063244] Key type .llcrypt registered [ 4377.175738] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4378.698627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4378.897953] alg: No test for adler32 (adler32-zlib) [ 4380.100068] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 4380.299080] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4381.960201] Key type lgssc registered [ 4383.756641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4393.983907] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4393.997332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4409.506831] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4414.627304] LustreError: 109357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 4441.312193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.999627] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4461.461235] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.039755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.561898] LustreError: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4469.582586] LustreError: 110383:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4469.596074] LustreError: 110383:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4469.606663] LustreError: 110383:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4469.620695] LustreError: 109363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785880706 with bad export cookie 4890392174177449136 [ 4469.630795] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4469.644048] LustreError: 109363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4469.799886] Lustre: server umount lustre-MDT0000 complete [ 4469.802835] LustreError: 110383:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4499.670379] Lustre: server umount lustre-OST0000 complete [ 4512.785142] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4530.229411] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4578.011182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4578.225194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4578.253630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4578.324351] Lustre: lustre-MDT0000: new disk, initializing [ 4578.408379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.431555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4583.624881] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4597.479277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.649550] Lustre: 113199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4597.717357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4597.723175] Lustre: Skipped 1 previous similar message [ 4597.834326] Lustre: lustre-MDT0001: new disk, initializing [ 4597.972425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4598.066217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4598.079634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4603.108424] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4614.554447] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4622.415574] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4630.888923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4631.364868] Lustre: lustre-OST0000: new disk, initializing [ 4631.372786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4631.396513] Lustre: 115178:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4631.574018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.366703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4639.383893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4639.635072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4640.003671] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4654.260862] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4661.027344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4661.296486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4666.905736] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4667.199923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4670.448227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.455876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4670.500620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4675.565919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4675.583983] Lustre: Skipped 1 previous similar message [ 4680.702593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4680.704936] Lustre: Skipped 1 previous similar message [ 4684.256689] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4684.731468] Lustre: server umount lustre-OST0000 complete [ 4690.304243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.308875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4690.335436] Lustre: Skipped 2 previous similar messages [ 4690.360722] Lustre: Skipped 2 previous similar messages [ 4695.523063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4695.531052] Lustre: Skipped 1 previous similar message [ 4697.411572] Lustre: server umount lustre-MDT0000 complete [ 4703.477890] LustreError: 113192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785880940 with bad export cookie 4890392174177449234 [ 4703.502986] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.511829] LustreError: 113192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4704.514525] Lustre: server umount lustre-MDT0001 complete [ 4718.755996] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:02:34 (1785880954) [ 4723.625655] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4745.482225] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 4757.761886] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4813.622720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.965129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4814.002597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4814.078752] Lustre: lustre-MDT0000: new disk, initializing [ 4814.285873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.313406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4820.481838] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4834.733591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.853199] Lustre: 120089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4834.901154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4834.905139] Lustre: Skipped 1 previous similar message [ 4834.987355] Lustre: lustre-MDT0001: new disk, initializing [ 4835.102079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4835.182947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.201234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4841.166748] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4853.977330] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.164417] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4874.669959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.971901] Lustre: lustre-OST0000: new disk, initializing [ 4874.980122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4874.988967] Lustre: 122058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4875.142976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.181797] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4884.663713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4884.693324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4884.937062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4895.472612] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4901.705329] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4901.964793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4907.814830] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4907.970151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4910.570049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.572259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4910.573722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4910.586491] Lustre: Skipped 1 previous similar message [ 4910.593587] LustreError: Skipped 1 previous similar message [ 4924.384172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4924.560862] Lustre: server umount lustre-OST0000 complete [ 4931.041586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4931.049422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.066121] Lustre: Skipped 1 previous similar message [ 4931.082373] Lustre: Skipped 6 previous similar messages [ 4943.328449] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4943.780859] Lustre: server umount lustre-MDT0000 complete [ 4946.401641] LustreError: 120102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.421193] LustreError: 120102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4947.781124] LustreError: 122068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881184 with bad export cookie 4890392174177450123 [ 4947.795564] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.068627] Lustre: server umount lustre-MDT0001 complete [ 4958.709270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4959.228552] LustreError: 124278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4959.316577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4964.332161] LustreError: 124279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.902968] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4969.449612] LustreError: 124278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.564269] LustreError: 124279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.824321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4981.630138] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4990.050858] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4997.936739] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5009.612933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5019.154222] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5030.375159] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5042.186105] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 5049.826738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.844415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.869173] Lustre: Skipped 1 previous similar message [ 5049.896635] Lustre: Skipped 5 previous similar messages [ 5055.597837] Lustre: server umount lustre-OST0000 complete [ 5063.649705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5063.664620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.660073] Lustre: server umount lustre-MDT0000 complete [ 5070.314378] LustreError: 124279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.357990] LustreError: 124279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5072.963356] LustreError: 124259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881310 with bad export cookie 4890392174177451005 [ 5072.969295] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.972155] LustreError: 124259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5073.611857] Lustre: server umount lustre-MDT0001 complete [ 5087.808474] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5091.812165] Key type lgssc unregistered [ 5092.150899] LNet: 127915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5092.157891] LNetError: 127915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5092.179451] LNet: Removed LNI 192.168.201.121@tcp [ 5092.981888] Key type .llcrypt unregistered [ 5092.983837] Key type ._llcrypt unregistered [ 5115.669225] Key type ._llcrypt registered [ 5115.671797] Key type .llcrypt registered [ 5115.809847] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5136.509519] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5139.363512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5139.410546] alg: No test for adler32 (adler32-zlib) [ 5140.768582] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 5141.326685] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5143.158145] Key type lgssc registered [ 5144.886721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5187.450363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5187.544492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.999445] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5189.083531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5189.286360] Lustre: lustre-MDT0000: new disk, initializing [ 5189.402352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.427616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5196.434705] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5208.934940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.060565] Lustre: 131071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5209.096761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5209.099869] Lustre: Skipped 1 previous similar message [ 5209.148949] Lustre: lustre-MDT0001: new disk, initializing [ 5209.220365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5209.264107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5209.275824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5215.180379] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5227.665621] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.833125] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5244.038795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5244.581410] Lustre: lustre-OST0000: new disk, initializing [ 5244.591537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5244.601229] Lustre: 133041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5244.778465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.223123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5251.242205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5251.348062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5253.010886] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5264.421906] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5270.082129] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5270.253145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5277.132423] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5277.379957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5282.285152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.286481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5282.290616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5282.329329] Lustre: Skipped 1 previous similar message [ 5282.366508] LustreError: Skipped 1 previous similar message [ 5286.450495] Lustre: server umount lustre-OST0000 complete [ 5292.513937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5292.529788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5292.539447] Lustre: Skipped 1 previous similar message [ 5297.635203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5297.648033] Lustre: Skipped 1 previous similar message [ 5302.759399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5302.772243] Lustre: Skipped 1 previous similar message [ 5306.337640] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5306.878874] Lustre: server umount lustre-MDT0000 complete [ 5311.688334] LustreError: 131066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881548 with bad export cookie 12402188233842474475 [ 5311.699680] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5312.359232] Lustre: server umount lustre-MDT0001 complete [ 5324.332773] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:12:40 (1785881560) [ 5332.552406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.109314] LustreError: 135455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.136814] LustreError: 135455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5333.256345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5338.594865] LustreError: 135456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.863299] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5343.714799] LustreError: 135455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.827870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.835865] LustreError: 135456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.326674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5355.457181] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5365.362968] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5373.384244] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5380.066688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5380.068494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.123105] Lustre: Skipped 1 previous similar message [ 5380.145257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5380.168113] Lustre: Skipped 1 previous similar message [ 5382.785539] Lustre: server umount lustre-MDT0000 complete [ 5385.185217] LustreError: 135455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.212137] LustreError: 135455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5386.915594] LustreError: 135437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881624 with bad export cookie 12402188233842475357 [ 5386.930201] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.934421] LustreError: 135437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5387.393758] Lustre: server umount lustre-MDT0001 complete [ 5399.271082] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:13:55 (1785881635) [ 5407.021890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5422.626773] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5427.744483] LustreError: 137874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 5453.537174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.024181] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5477.106868] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5479.351555] Lustre: server umount lustre-OST0000 complete [ 5489.068387] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:15:25 (1785881725) [ 5498.489383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5499.198709] LustreError: 139324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.300220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.910711] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5516.143421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.613561] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5532.768684] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5540.497231] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5548.121872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5548.638766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.646631] Lustre: Skipped 1 previous similar message [ 5557.467266] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5568.812466] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5583.370698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5583.409717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.438376] Lustre: Skipped 1 previous similar message [ 5583.446695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5586.654355] Lustre: server umount lustre-MDT0000 complete [ 5589.989425] LustreError: 139325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.016605] LustreError: 139325:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5592.510775] LustreError: 139306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881829 with bad export cookie 12402188233842476022 [ 5592.513569] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5592.516202] LustreError: 139306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5593.295960] Lustre: server umount lustre-MDT0001 complete [ 5609.358340] Lustre: server umount lustre-OST0000 complete [ 5620.212377] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 18:17:36 (1785881856) [ 5628.421415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5629.111057] LustreError: 142981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.217820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.937225] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5645.928670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.340517] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5663.702614] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5670.952368] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5681.270432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5693.517793] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5699.266376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5707.547891] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5714.149452] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5714.457223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5722.494963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5722.851088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5727.202116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5727.210656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.232153] Lustre: Skipped 2 previous similar messages [ 5727.243663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.257102] Lustre: Skipped 3 previous similar messages [ 5731.964501] Lustre: server umount lustre-OST0000 complete [ 5738.466034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5738.476547] LustreError: Skipped 1 previous similar message [ 5738.478456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.500205] Lustre: Skipped 1 previous similar message [ 5743.900548] Lustre: server umount lustre-MDT0000 complete [ 5745.633905] LustreError: 142981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5745.670031] LustreError: 142981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5748.968249] LustreError: 142962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785881986 with bad export cookie 12402188233842477086 [ 5748.985846] LustreError: 142962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5748.987130] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5749.725244] Lustre: server umount lustre-MDT0001 complete [ 5761.145571] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 18:19:57 (1785881997) [ 5772.170981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5788.001107] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5793.120488] LustreError: 147146:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 5818.848678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.857202] Lustre: Skipped 2 previous similar messages [ 5826.188411] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5837.530601] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5845.679297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5846.380046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5852.618345] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5862.526815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.947947] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5878.445796] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5885.496644] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5893.227626] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5893.685329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5900.408865] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5900.731298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5903.843837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5903.845636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.868630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.890294] Lustre: Skipped 2 previous similar messages [ 5903.909093] Lustre: Skipped 5 previous similar messages [ 5909.837683] Lustre: server umount lustre-MDT0000 complete [ 5914.097949] LustreError: 147172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.134969] LustreError: 147172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5914.664782] LustreError: 147154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785882151 with bad export cookie 12402188233842478115 [ 5914.679024] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5914.681016] LustreError: 147154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5915.206391] Lustre: server umount lustre-MDT0001 complete [ 5930.122472] Lustre: server umount lustre-OST0000 complete [ 5941.341447] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 18:22:57 (1785882177) [ 5949.307741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.960993] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5970.081252] LustreError: 151347:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 6004.862845] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6015.392967] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6024.538569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.377148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 6031.038940] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6040.958084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.037551] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6057.167689] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6065.869196] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6078.891080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.304975] Lustre: lustre-OST0001: new disk, initializing [ 6079.314960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6079.347685] Lustre: 154221:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6079.572060] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6079.583305] Lustre: Skipped 5 previous similar messages [ 6085.250371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6085.270670] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6085.414653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6090.269729] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6102.924778] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6109.181464] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6109.437139] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6116.152658] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6116.485651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6121.450417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.469859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6121.471563] Lustre: Skipped 2 previous similar messages [ 6121.491122] Lustre: Skipped 4 previous similar messages [ 6125.446667] Lustre: server umount lustre-OST0000 complete [ 6131.702860] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.896793] Lustre: server umount lustre-OST0001 complete [ 6148.672736] Lustre: server umount lustre-MDT0000 complete [ 6153.655375] LustreError: 151354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785882390 with bad export cookie 12402188233842479067 [ 6153.661957] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.671090] LustreError: 151354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6154.671353] Lustre: server umount lustre-MDT0001 complete [ 6196.517062] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 18:27:12 (1785882432) [ 6199.556162] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6203.316433] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6204.927644] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 18:27:21 (1785882441) [ 6212.993342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6213.404128] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6213.470186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6219.664637] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6230.891036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.037416] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6231.095969] Lustre: 157789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6238.567796] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6248.355966] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6257.955707] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6267.083567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6267.338076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6273.242635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6276.169951] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6291.178359] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.699353] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6297.963368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6303.550486] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6303.722499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6309.348696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.360891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.367368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6309.367377] Lustre: Skipped 6 previous similar messages [ 6309.381576] Lustre: Skipped 5 previous similar messages [ 6309.453353] LustreError: Skipped 2 previous similar messages [ 6312.774574] Lustre: server umount lustre-OST0000 complete [ 6341.647217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.458361] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.506280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.511369] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6349.549642] Lustre: Skipped 1 previous similar message [ 6349.928943] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6361.369678] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6369.559182] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6369.863166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6377.752961] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6378.121638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6385.101571] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f6c843aa800.ost_server_uuid 50 [ 6386.917637] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f6c843aa800.ost_server_uuid in IDLE state after 0 sec [ 6387.283559] LustreError: 157797:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.21@tcp) returned error from blocking AST (req@00000000f445e793 x1872632352122368 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000002099093f/0xac1d6c7d33b5d935 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.21@tcp remote: 0x6b9e09a3d9145d63 expref: 5 pid: 157797 timeout: 6487 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6387.338314] LustreError: lustre-MDT0000: A client on nid 192.168.201.21@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6387.362184] LustreError: 157786:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.21@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000002099093f/0xac1d6c7d33b5d935 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.21@tcp remote: 0x6b9e09a3d9145d63 expref: 6 pid: 157797 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6398.538540] Lustre: server umount lustre-OST0000 complete [ 6405.601491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6414.796182] Lustre: server umount lustre-MDT0000 complete [ 6416.364381] LustreError: 157802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.403736] LustreError: 157802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 6417.192650] Lustre: 129298:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785882638/real 1785882638] req@0000000010be436c x1872632352126208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785882654 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6420.192686] LustreError: 162955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785882657 with bad export cookie 12402188233842480243 [ 6420.197709] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.201810] LustreError: 162955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6427.219361] Lustre: server umount lustre-MDT0001 complete [ 6442.770641] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6446.579392] Key type lgssc unregistered [ 6447.070078] LNet: 163915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6447.076712] LNetError: 163915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6447.103206] LNet: Removed LNI 192.168.201.121@tcp [ 6448.215429] Key type .llcrypt unregistered [ 6448.219721] Key type ._llcrypt unregistered [ 6478.494602] Key type ._llcrypt registered [ 6478.497664] Key type .llcrypt registered [ 6478.591761] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6479.723549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6479.994922] alg: No test for adler32 (adler32-zlib) [ 6481.356594] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 6481.676694] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6483.520160] Key type lgssc registered [ 6486.140505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6498.192399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6498.255445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.991268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.600466] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6510.565475] LustreError: 165331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.597893] LustreError: 165331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6515.552297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.682557] LustreError: 165332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.117901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6522.867837] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6532.908118] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6540.076906] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.165533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.688837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6557.385877] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6566.798202] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.248291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6577.805123] Lustre: Failing over lustre-MDT0000 [ 6578.269355] LustreError: 165326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.384041] Lustre: server umount lustre-MDT0000 complete [ 6580.716283] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.718322] LustreError: 165327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6580.735130] Lustre: Skipped 2 previous similar messages [ 6580.762595] LustreError: 165327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6585.825899] LustreError: 165326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.846399] LustreError: 165326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6591.492855] LustreError: 165312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.21@tcp arrived at 1785882828 with bad export cookie 4438356073908606344 [ 6591.512337] LustreError: 165312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6596.069492] LustreError: 166259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.070602] Lustre: 164642:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785882817/real 1785882817] req@00000000133920a3 x1872633756940160/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785882833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6596.086916] LustreError: 166259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6596.156481] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6602.002771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.306452] LustreError: 164638:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000941e4c1d x1872633756945024/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6607.041841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.215315] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6612.467589] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6612.479175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6615.238471] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6619.075460] Lustre: lustre-MDT0000: Denying connection for new client 1beac35d-ddea-4b88-b283-3969922b6a1f (at 192.168.201.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 6624.312927] Lustre: lustre-MDT0000: Denying connection for new client 1beac35d-ddea-4b88-b283-3969922b6a1f (at 192.168.201.21@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 6632.932983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6632.935563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.943812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6632.969447] Lustre: Skipped 1 previous similar message [ 6634.664838] Lustre: server umount lustre-OST0000 complete [ 6639.987552] LustreError: 169063:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6639.995349] Lustre: 168412:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6640.004300] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6640.038761] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6640.054863] LustreError: 168412:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000941e4c1d x1872633756962560/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6640.070541] LustreError: 168412:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6640.079487] LustreError: 168412:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6640.091566] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 6640.706395] Lustre: server umount lustre-MDT0000 complete [ 6643.175903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6643.217312] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.237561] LustreError: 165328:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6646.802942] LustreError: 164638:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 6646.807653] LustreError: 167176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785882883 with bad export cookie 4438356073908606666 [ 6646.813634] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.863037] LustreError: 167176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6653.403847] Lustre: server umount lustre-MDT0001 complete [ 6672.949548] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6678.395067] Key type lgssc unregistered [ 6678.917654] LNet: 169842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6678.940556] LNetError: 169842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6679.007642] LNet: Removed LNI 192.168.201.121@tcp [ 6680.368600] Key type .llcrypt unregistered [ 6680.373604] Key type ._llcrypt unregistered [ 6715.852570] Key type ._llcrypt registered [ 6715.866140] Key type .llcrypt registered [ 6715.982206] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6718.075549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6718.112386] alg: No test for adler32 (adler32-zlib) [ 6719.528816] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 6719.809715] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6721.537517] Key type lgssc registered [ 6723.663535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6737.118278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6737.143824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.832958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.615229] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6748.145312] LustreError: 171264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.182455] LustreError: 171264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6753.250342] LustreError: 171265:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.463435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.104201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6764.631180] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6777.143606] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6784.879495] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6794.986824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6795.825952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.298989] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6808.125168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6816.258541] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6828.519476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6828.522207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.539921] LustreError: Skipped 1 previous similar message [ 6828.548748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6828.566508] Lustre: Skipped 1 previous similar message [ 6828.611440] Lustre: Skipped 1 previous similar message [ 6832.197153] Lustre: server umount lustre-OST0000 complete [ 6840.292604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6840.311547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.350770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.385646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.387028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.872186] Lustre: server umount lustre-MDT0000 complete [ 6848.547665] LustreError: 171264:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.576051] LustreError: 171264:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6852.387657] LustreError: 174051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785883089 with bad export cookie 8416827476758545734 [ 6852.394878] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6852.408686] LustreError: 174051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6853.487710] Lustre: server umount lustre-MDT0001 complete [ 6873.552334] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 6877.815942] Key type lgssc unregistered [ 6878.265380] LNet: 174822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6878.274390] LNetError: 174822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6878.293623] LNet: Removed LNI 192.168.201.121@tcp [ 6879.100514] Key type .llcrypt unregistered [ 6879.102778] Key type ._llcrypt unregistered [ 6926.238874] Key type ._llcrypt registered [ 6926.241631] Key type .llcrypt registered [ 6926.374697] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 6927.503249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6927.727680] alg: No test for adler32 (adler32-zlib) [ 6928.789912] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 6929.063845] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 6930.736173] Key type lgssc registered [ 6932.633934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6944.640667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6944.711928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6946.707457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6955.439397] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6956.035669] LustreError: 176525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.079341] LustreError: 176525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6961.122177] LustreError: 176526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.247857] LustreError: 176525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.722868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6967.952620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6974.270148] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 6985.595585] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6993.686953] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.300702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.677470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7010.191924] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7015.992389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 7018.813440] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7028.193785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7028.201949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.219534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.276224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.290782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.297685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.328438] Lustre: Skipped 1 previous similar message [ 7033.004435] Lustre: server umount lustre-OST0000 complete [ 7039.458859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7039.471787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7039.498764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.505356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.506272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.552875] Lustre: Skipped 1 previous similar message [ 7046.628096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.648021] Lustre: Skipped 1 previous similar message [ 7053.282417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7053.775539] Lustre: server umount lustre-MDT0000 complete [ 7056.869183] LustreError: 177371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7056.924200] LustreError: 177371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7059.802060] LustreError: 176508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785883296 with bad export cookie 10141215967609200067 [ 7059.808023] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.823950] LustreError: 176508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.515787] Lustre: server umount lustre-MDT0001 complete [ 7080.023525] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7084.005677] Key type lgssc unregistered [ 7084.585505] LNet: 180090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7084.599490] LNetError: 180090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7084.628171] LNet: Removed LNI 192.168.201.121@tcp [ 7086.405884] Key type .llcrypt unregistered [ 7086.416061] Key type ._llcrypt unregistered [ 7120.383767] Key type ._llcrypt registered [ 7120.386731] Key type .llcrypt registered [ 7120.595516] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7122.297872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.635262] alg: No test for adler32 (adler32-zlib) [ 7123.983793] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 7124.456527] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7126.256922] Key type lgssc registered [ 7129.069643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7151.475126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7151.527353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.375696] Lustre: *** cfs_fail_loc=135, val=0*** [ 7153.604340] LustreError: 181619:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7153.628550] LustreError: 181619:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -2 [ 7153.645795] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7153.645795] [ 7153.682199] LustreError: MGC192.168.201.121@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7153.706598] LustreError: 181581:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7153.725112] LustreError: 181581:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7153.728240] LustreError: 181581:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7153.755200] LustreError: 181581:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7154.014544] Lustre: server umount lustre-MDT0000 complete [ 7154.024393] LustreError: 181581:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7165.216489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7166.495132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7174.584283] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7176.674661] LustreError: 181991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7176.698542] LustreError: 181991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7181.798334] LustreError: 181992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.610589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7185.004290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7190.403177] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7202.898900] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7213.032604] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7220.706322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7220.724179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7220.753209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7221.735511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7221.747146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7225.444771] Lustre: server umount lustre-MDT0000 complete [ 7226.854721] LustreError: 181988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.886376] LustreError: 181988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7228.906655] LustreError: 182934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7228.943389] LustreError: 182934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7230.794763] LustreError: 181973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785883467 with bad export cookie 1002283452105500062 [ 7230.830177] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.828534] Lustre: server umount lustre-MDT0001 complete [ 7252.357784] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7257.464466] Key type lgssc unregistered [ 7258.111676] LNet: 184350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7258.123920] LNetError: 184350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7258.155890] LNet: Removed LNI 192.168.201.121@tcp [ 7260.116858] Key type .llcrypt unregistered [ 7260.121444] Key type ._llcrypt unregistered [ 7293.996161] Key type ._llcrypt registered [ 7293.997539] Key type .llcrypt registered [ 7294.195770] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7294.660776] Key type .llcrypt unregistered [ 7294.664376] Key type ._llcrypt unregistered [ 7318.018447] Key type ._llcrypt registered [ 7318.020608] Key type .llcrypt registered [ 7318.132278] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7319.860178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7320.115046] alg: No test for adler32 (adler32-zlib) [ 7321.417752] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 7321.779037] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7323.448175] Key type lgssc registered [ 7325.364616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7336.924497] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7337.012170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.864438] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7357.985098] LustreError: 186488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 7384.736298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.440911] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7409.111182] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7417.817866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.214817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.287719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7424.484197] LustreError: 186513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7424.513315] LustreError: 186513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7424.537843] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7429.617737] LustreError: 187527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7429.642823] LustreError: 187527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7434.730262] LustreError: 187529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.763215] LustreError: 187529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7438.188864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.405037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.139808] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7457.207996] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7465.068853] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7468.266616] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7484.405234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7484.450904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7489.518023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7489.530779] Lustre: Skipped 2 previous similar messages [ 7494.644735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7494.660429] Lustre: Skipped 1 previous similar message [ 7496.165160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7496.584772] Lustre: server umount lustre-OST0000 complete [ 7504.873351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7504.890320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.917373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7504.975698] Lustre: Skipped 2 previous similar messages [ 7509.999294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7510.004297] Lustre: Skipped 2 previous similar messages [ 7511.232897] Lustre: server umount lustre-MDT0000 complete [ 7515.160783] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7515.212414] LustreError: 187533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7520.233442] LustreError: 187527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7520.244552] LustreError: 188272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785883757 with bad export cookie 12896489858903307421 [ 7520.283302] LustreError: 187527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7520.343640] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7520.347715] LustreError: 188272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7522.880858] Lustre: server umount lustre-MDT0001 complete [ 7539.872247] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7543.346646] Key type lgssc unregistered [ 7543.628270] LNet: 190467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7543.636493] LNetError: 190467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7543.655855] LNet: Removed LNI 192.168.201.121@tcp [ 7545.052852] Key type .llcrypt unregistered [ 7545.058061] Key type ._llcrypt unregistered [ 7580.145823] Key type ._llcrypt registered [ 7580.147522] Key type .llcrypt registered [ 7580.336683] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7582.563381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7582.616160] alg: No test for adler32 (adler32-zlib) [ 7583.839286] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 7584.483231] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7586.289539] Key type lgssc registered [ 7588.518496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7599.887123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7599.926024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.823757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7608.175831] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7611.376560] LustreError: 191882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7612.069512] LustreError: 191882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7616.505846] LustreError: 191883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7621.602698] LustreError: 191882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7627.044248] LustreError: 191883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.908997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.051753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.018506] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7649.159410] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7658.794400] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7667.766618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7668.141828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.268744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7678.055340] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7690.716393] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7699.417629] Lustre: Failing over lustre-MDT0000 [ 7699.948606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.992991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.375433] Lustre: server umount lustre-MDT0000 complete [ 7703.139325] LustreError: 191879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.349545] LustreError: 191877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.365083] LustreError: 191877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7721.443511] Lustre: 191193:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785883942/real 1785883942] req@00000000b68df828 x1872634913523584/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785883958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7721.511240] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7725.441455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.726857] LustreError: 191879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.771573] LustreError: 191879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 7732.253644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.360493] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7734.862669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7737.332330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7737.350645] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7737.396179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7739.358533] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7745.072095] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7752.768364] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7766.501170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7766.528550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.563681] Lustre: Skipped 2 previous similar messages [ 7766.567345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7766.569983] Lustre: Skipped 2 previous similar messages [ 7768.043234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.045790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.564106] Lustre: server umount lustre-OST0000 complete [ 7778.274737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7778.276618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.290713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.308177] Lustre: Skipped 1 previous similar message [ 7778.322924] Lustre: Skipped 2 previous similar messages [ 7783.405094] LustreError: 191883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.449764] LustreError: 191883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7785.579301] Lustre: server umount lustre-MDT0000 complete [ 7790.751585] LustreError: 191864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785884027 with bad export cookie 1366156903482386769 [ 7790.758934] LustreError: 191864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7790.760683] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7791.254369] Lustre: server umount lustre-MDT0001 complete [ 7812.856404] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7817.050426] Key type lgssc unregistered [ 7817.621546] LNet: 196760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7817.632783] LNetError: 196760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7817.660210] LNet: Removed LNI 192.168.201.121@tcp [ 7819.142567] Key type .llcrypt unregistered [ 7819.146812] Key type ._llcrypt unregistered [ 7854.644566] Key type ._llcrypt registered [ 7854.647495] Key type .llcrypt registered [ 7854.769286] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7856.623894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7856.810825] alg: No test for adler32 (adler32-zlib) [ 7858.204142] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 7858.720460] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7860.656168] Key type lgssc registered [ 7863.171611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7874.851686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7874.891500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.817984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.291523] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7886.306423] LustreError: 198177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.325846] LustreError: 198177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7891.426388] LustreError: 198178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.537101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7896.121237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7903.196841] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7916.366061] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7924.343548] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7933.457112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.122685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.642789] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7947.413712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 7954.860492] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.056388] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7963.985952] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 7982.991523] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7992.801433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7992.815238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.834155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7993.829927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7993.833146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7993.837114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.877756] Lustre: Skipped 1 previous similar message [ 7995.418763] Lustre: server umount lustre-OST0000 complete [ 8003.577602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8003.596547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.628444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8007.955682] Lustre: server umount lustre-MDT0000 complete [ 8009.190672] LustreError: 198177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.365862] LustreError: 198159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785884250 with bad export cookie 9384486670965170739 [ 8013.377363] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8013.378967] LustreError: 198159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8014.131457] Lustre: server umount lustre-MDT0001 complete [ 8031.483062] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8036.274063] Key type lgssc unregistered [ 8036.839603] LNet: 202047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8036.850027] LNetError: 202047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8036.897841] LNet: Removed LNI 192.168.201.121@tcp [ 8038.512225] Key type .llcrypt unregistered [ 8038.516895] Key type ._llcrypt unregistered [ 8071.083573] Key type ._llcrypt registered [ 8071.085015] Key type .llcrypt registered [ 8071.211793] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8074.159648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8074.232365] alg: No test for adler32 (adler32-zlib) [ 8075.707563] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 8076.270187] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8078.072162] Key type lgssc registered [ 8079.863317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8091.740944] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8091.772425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8093.844816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8100.411729] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8103.395211] LustreError: 203315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.441888] LustreError: 203315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8108.521730] LustreError: 203316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.299383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.947186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.557453] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8128.599844] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8137.542121] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8146.839812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.287720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.974717] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8163.742926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8167.273715] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.708273] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8187.713495] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8199.730209] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8205.378185] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8223.201301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8223.212689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8223.249368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.254188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.255906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.298962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.169930] Lustre: server umount lustre-OST0000 complete [ 8234.977162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8234.988378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.012195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8235.014855] Lustre: Skipped 1 previous similar message [ 8240.610602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8240.623818] Lustre: Skipped 3 previous similar messages [ 8241.293094] Lustre: server umount lustre-MDT0000 complete [ 8245.735129] LustreError: 203315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8245.755673] LustreError: 203315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8246.945810] LustreError: 203295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785884484 with bad export cookie 545493125660535749 [ 8246.953255] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8246.971097] LustreError: 203295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8247.378985] Lustre: server umount lustre-MDT0001 complete [ 8263.322794] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8267.624861] Key type lgssc unregistered [ 8268.051213] LNet: 208117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8268.055483] LNetError: 208117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8268.090423] LNet: Removed LNI 192.168.201.121@tcp [ 8269.429243] Key type .llcrypt unregistered [ 8269.430985] Key type ._llcrypt unregistered [ 8301.725714] Key type ._llcrypt registered [ 8301.730477] Key type .llcrypt registered [ 8301.902164] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8303.365164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8303.417067] alg: No test for adler32 (adler32-zlib) [ 8304.732722] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 8305.252503] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8307.052356] Key type lgssc registered [ 8308.949227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8320.388977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8320.427941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.203102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.681963] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8332.777115] LustreError: 209535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.805944] LustreError: 209535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8337.891659] LustreError: 209536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.922131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.621851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.670201] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8367.738835] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8376.625995] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8387.433149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.211183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.519611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8401.951471] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8416.394988] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8439.779498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8439.794077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.814755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8441.828199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.831566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8441.876787] Lustre: Skipped 1 previous similar message [ 8443.787650] Lustre: server umount lustre-OST0000 complete [ 8448.997675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8449.008697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.045051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.067121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8452.067676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.892675] Lustre: server umount lustre-MDT0000 complete [ 8457.187488] LustreError: 209536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.224485] LustreError: 209536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8459.184818] LustreError: 212474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785884696 with bad export cookie 12155454052883866084 [ 8459.192308] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.198486] LustreError: 212474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8459.735209] Lustre: server umount lustre-MDT0001 complete [ 8476.025281] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8480.403183] Key type lgssc unregistered [ 8480.752612] LNet: 213230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8480.766912] LNetError: 213230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8481.841105] LNet: Removed LNI 192.168.201.121@tcp [ 8483.449369] Key type .llcrypt unregistered [ 8483.454636] Key type ._llcrypt unregistered [ 8512.333956] Key type ._llcrypt registered [ 8512.337537] Key type .llcrypt registered [ 8512.514116] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8514.365320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8514.610881] alg: No test for adler32 (adler32-zlib) [ 8515.658669] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 8515.888704] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8517.528174] Key type lgssc registered [ 8519.198655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8529.288844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8529.317405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8531.014350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8537.265024] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8541.155564] LustreError: 214499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.176884] LustreError: 214499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8546.274588] LustreError: 214500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8549.174302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.507949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8555.529614] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8564.309514] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8573.813183] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8583.776603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8584.454005] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8591.944636] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8601.929255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8602.973882] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8625.633604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8625.638499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8625.646400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8627.681636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8627.688175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8627.698117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8627.724490] Lustre: Skipped 1 previous similar message [ 8630.594323] Lustre: server umount lustre-OST0000 complete [ 8636.385659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8636.400967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.427355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8642.025529] Lustre: server umount lustre-MDT0000 complete [ 8643.047763] LustreError: 215228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.098716] LustreError: 215228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8648.167715] LustreError: 216038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8648.187641] LustreError: 216038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8648.367268] LustreError: 214479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785884885 with bad export cookie 16625247107801983012 [ 8648.379939] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8650.108555] Lustre: server umount lustre-MDT0001 complete [ 8675.101689] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8682.064804] Key type lgssc unregistered [ 8682.449144] LNet: 218215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8682.467932] LNetError: 218215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8682.483787] LNet: Removed LNI 192.168.201.121@tcp [ 8683.804375] Key type .llcrypt unregistered [ 8683.809084] Key type ._llcrypt unregistered [ 8713.718945] Key type ._llcrypt registered [ 8713.721238] Key type .llcrypt registered [ 8713.939567] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8715.397914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8715.521750] alg: No test for adler32 (adler32-zlib) [ 8716.705689] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 8717.030473] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8718.760185] Key type lgssc registered [ 8721.684414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8734.027845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8734.112353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.806142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.794984] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8744.951407] LustreError: 219485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.978414] LustreError: 219485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8750.050100] LustreError: 219486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.395442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.304803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8762.595516] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8775.952842] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8786.122490] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8796.410706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8797.465162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.807360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8807.953395] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8818.223136] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8829.577364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.661042] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8829.758813] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8837.082553] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8848.852258] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8851.749597] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8863.711130] Lustre: Permanently deactivating lustre-OST0001 [ 8863.719267] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8870.522507] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8883.575352] Lustre: Permanently reactivating lustre-OST0001 [ 8883.598956] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8883.606565] Lustre: Skipped 2 previous similar messages [ 8889.457985] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8889.486304] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8889.494239] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8889.507048] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8889.564270] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8894.951404] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.954197] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.961298] LustreError: Skipped 1 previous similar message [ 8894.963263] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8895.021068] Lustre: Skipped 2 previous similar messages [ 8900.076971] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8900.087517] Lustre: Skipped 2 previous similar messages [ 8900.719486] Lustre: server umount lustre-OST0001 complete [ 8905.194157] LustreError: 221308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8905.216990] LustreError: 221308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8908.257502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8908.261831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.272267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8909.291722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8910.358227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8910.373163] Lustre: Skipped 2 previous similar messages [ 8910.386209] LustreError: 223585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.463637] LustreError: 223585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8912.979882] Lustre: server umount lustre-OST0000 complete [ 8919.009700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8919.024896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8919.042627] Lustre: Skipped 1 previous similar message [ 8919.050190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.785504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8930.805777] Lustre: Skipped 5 previous similar messages [ 8933.346165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8933.730966] Lustre: server umount lustre-MDT0000 complete [ 8935.907018] LustreError: 221612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.926956] LustreError: 221612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8939.486537] LustreError: 222326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785885176 with bad export cookie 1690424557714586558 [ 8939.496734] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8939.507795] LustreError: 222326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8940.435964] Lustre: server umount lustre-MDT0001 complete [ 8957.777535] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8962.433404] Key type lgssc unregistered [ 8963.034553] LNet: 224544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8963.045935] LNetError: 224544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8963.079381] LNet: Removed LNI 192.168.201.121@tcp [ 8964.339463] Key type .llcrypt unregistered [ 8964.344480] Key type ._llcrypt unregistered [ 9015.380312] Key type ._llcrypt registered [ 9015.388325] Key type .llcrypt registered [ 9015.527902] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9017.291699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9017.386578] alg: No test for adler32 (adler32-zlib) [ 9018.550276] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 9018.910884] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9020.648203] Key type lgssc registered [ 9022.201687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9033.148912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9033.192215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.637166] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9034.736684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9035.024112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9041.904459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9053.715966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.882213] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9053.947554] Lustre: 226754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9054.440276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9060.642700] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9070.029775] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9077.555538] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9088.174557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9088.631939] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9088.978859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9097.070353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 9100.627231] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9114.160204] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9117.626750] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 9123.893871] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9131.470639] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 9136.816420] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9149.853331] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9149.870174] Lustre: Skipped 1 previous similar message [ 9168.896371] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 9168.899932] Lustre: Skipped 1 previous similar message [ 9202.782777] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 9202.790929] Lustre: Skipped 3 previous similar messages [ 9267.986937] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9267.995328] Lustre: Skipped 8 previous similar messages [ 9296.865958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9296.878532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9296.881610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9296.898970] Lustre: Skipped 1 previous similar message [ 9296.906842] Lustre: Skipped 1 previous similar message [ 9301.987506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9301.991392] Lustre: Skipped 1 previous similar message [ 9307.109180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9307.113392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.133885] Lustre: Skipped 1 previous similar message [ 9307.338216] Lustre: server umount lustre-OST0000 complete [ 9315.809424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9315.813720] LustreError: Skipped 1 previous similar message [ 9315.816657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.825610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9317.363774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.218857] Lustre: server umount lustre-MDT0000 complete [ 9324.287258] LustreError: 229769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785885561 with bad export cookie 1124077171340716105 [ 9324.288949] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9324.307557] LustreError: 229769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9324.777582] Lustre: server umount lustre-MDT0001 complete [ 9338.768748] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9343.094904] Key type lgssc unregistered [ 9343.458150] LNet: 231573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9343.467269] LNetError: 231573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9343.509365] LNet: Removed LNI 192.168.201.121@tcp [ 9344.589512] Key type .llcrypt unregistered [ 9344.591856] Key type ._llcrypt unregistered [ 9384.058742] Key type ._llcrypt registered [ 9384.063778] Key type .llcrypt registered [ 9384.223947] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9385.546084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9385.655483] alg: No test for adler32 (adler32-zlib) [ 9387.157368] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 9387.706809] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9389.616226] Key type lgssc registered [ 9391.550376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9406.288477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9406.320304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.211953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9413.940461] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9418.737479] LustreError: 232995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9418.763928] LustreError: 232995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9423.843246] LustreError: 232996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9424.704488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.431677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9432.830579] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9444.508484] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9451.497039] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9462.137337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9462.639584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.852952] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9478.012245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9480.787856] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9498.597131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9498.599503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9498.620944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9498.638628] Lustre: Skipped 1 previous similar message [ 9503.866650] Lustre: server umount lustre-OST0000 complete [ 9512.418408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9512.421353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.449338] Lustre: Skipped 1 previous similar message [ 9512.456763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9513.953377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9513.954354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9513.983572] Lustre: Skipped 1 previous similar message [ 9515.409837] Lustre: server umount lustre-MDT0000 complete [ 9518.114483] LustreError: 233727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9518.133260] LustreError: 233727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9519.655555] LustreError: 232977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785885756 with bad export cookie 14755215496013718954 [ 9519.664806] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9520.085193] Lustre: server umount lustre-MDT0001 complete [ 9532.529695] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9536.050915] Key type lgssc unregistered [ 9536.477300] LNet: 236692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9536.484266] LNetError: 236692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9536.510430] LNet: Removed LNI 192.168.201.121@tcp [ 9537.797019] Key type .llcrypt unregistered [ 9537.798702] Key type ._llcrypt unregistered [ 9579.629373] Key type ._llcrypt registered [ 9579.630806] Key type .llcrypt registered [ 9579.900101] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9582.392461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9582.498865] alg: No test for adler32 (adler32-zlib) [ 9584.319159] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 9584.920239] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9586.801270] Key type lgssc registered [ 9589.081605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9864.056033] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9866.730861] Key type lgssc unregistered [ 9867.104589] LNet: 238734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9867.111585] LNetError: 238734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9867.136389] LNet: Removed LNI 192.168.201.121@tcp [ 9868.333929] Key type .llcrypt unregistered [ 9868.335450] Key type ._llcrypt unregistered [ 9910.885370] Key type ._llcrypt registered [ 9910.891706] Key type .llcrypt registered [ 9911.087344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 9936.951274] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9938.682485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9938.709644] alg: No test for adler32 (adler32-zlib) [ 9939.933910] Lustre: Lustre: Build Version: 2.17.56_50_g55fd350 [ 9940.278489] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9942.448573] Key type lgssc registered [ 9944.840587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9984.840130] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9628 sec ========== 19:30:20 (1785886220) [ 9986.864093] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:30:23 (1785886223) === [ 9992.220600] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:30:28 (1785886228) === [10042.704698] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [10047.786586] Key type lgssc unregistered [10048.175757] LNet: 243994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10048.183316] LNetError: 243994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10048.207825] LNet: Removed LNI 192.168.201.121@tcp [10049.915442] Key type .llcrypt unregistered [10049.920856] Key type ._llcrypt unregistered