[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 783671130 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003325] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008451] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012047] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.023380] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028031] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029203] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031108] *** VALIDATE tmpfs *** [ 0.033384] *** VALIDATE proc *** [ 0.034231] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038134] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040026] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044231] debug: unmapping init [mem 0xffffffff8da59000-0xffffffff8da60fff] [ 0.046247] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049010] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052010] ... max period: 00007fffffffffff [ 0.053010] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055317] rcu: Hierarchical SRCU implementation. [ 0.057862] smp: Bringing up secondary CPUs ... [ 0.058640] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.067079] smp: Brought up 1 node, 4 CPUs [ 0.069013] smpboot: Max logical packages: 1 [ 0.070013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.107203] node 0 deferred pages initialised in 35ms [ 0.112588] devtmpfs: initialized [ 0.114426] x86/mm: Memory block size: 128MB [ 0.117967] gcov: version magic: 0x41383552 [ 0.119667] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.128022] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.133701] pinctrl core: initialized pinctrl subsystem [ 0.136225] [ 0.136977] ************************************************************* [ 0.141027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144022] ** ** [ 0.147022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150022] ** ** [ 0.154022] ** This means that this kernel is built to expose internal ** [ 0.156011] ** IOMMU data structures, which may compromise security on ** [ 0.160012] ** your system. ** [ 0.164027] ** ** [ 0.167010] ** If you see this message and you are not debugging the ** [ 0.170014] ** kernel, report this immediately to your vendor! ** [ 0.175013] ** ** [ 0.176010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178010] ************************************************************* [ 0.180711] NET: Registered protocol family 16 [ 0.182481] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.188108] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.191132] cpuidle: using governor menu [ 0.193155] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195546] PCI: Using configuration type 1 for base access [ 0.198131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212397] cryptd: max_cpu_qlen set to 1000 [ 0.214000] ACPI: Added _OSI(Module Device) [ 0.214012] ACPI: Added _OSI(Processor Device) [ 0.217047] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.225878] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.232573] ACPI: Interpreter enabled [ 0.235148] ACPI: PM: (supports S0 S3 S4 S5) [ 0.236096] ACPI: Using IOAPIC for interrupt routing [ 0.239089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243459] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255546] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264107] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269650] acpiphp: Slot [2] registered [ 0.271099] acpiphp: Slot [5] registered [ 0.273161] acpiphp: Slot [6] registered [ 0.275123] acpiphp: Slot [7] registered [ 0.276000] acpiphp: Slot [8] registered [ 0.276000] acpiphp: Slot [9] registered [ 0.276131] acpiphp: Slot [10] registered [ 0.277000] acpiphp: Slot [3] registered [ 0.280129] acpiphp: Slot [4] registered [ 0.281079] acpiphp: Slot [11] registered [ 0.282077] acpiphp: Slot [12] registered [ 0.284088] acpiphp: Slot [13] registered [ 0.285087] acpiphp: Slot [14] registered [ 0.287110] acpiphp: Slot [15] registered [ 0.288121] acpiphp: Slot [16] registered [ 0.290106] acpiphp: Slot [17] registered [ 0.291163] acpiphp: Slot [18] registered [ 0.293108] acpiphp: Slot [19] registered [ 0.294093] acpiphp: Slot [20] registered [ 0.296162] acpiphp: Slot [21] registered [ 0.297138] acpiphp: Slot [22] registered [ 0.299114] acpiphp: Slot [23] registered [ 0.300102] acpiphp: Slot [24] registered [ 0.302189] acpiphp: Slot [25] registered [ 0.304131] acpiphp: Slot [26] registered [ 0.305119] acpiphp: Slot [27] registered [ 0.307120] acpiphp: Slot [28] registered [ 0.309129] acpiphp: Slot [29] registered [ 0.310087] acpiphp: Slot [30] registered [ 0.312078] acpiphp: Slot [31] registered [ 0.313057] PCI host bridge to bus 0000:00 [ 0.314021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.317043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319058] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.327031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.329259] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332067] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335283] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344875] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.350575] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.353017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.356017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363527] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365839] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371877] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.389014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.394014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401378] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.410097] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.487836] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.525017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.588000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.595021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.606019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646435] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.664016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.678014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.726490] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.744019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.764033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.792021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.808989] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.819018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.830014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.850018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.864505] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.868421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.870340] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.873400] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.876208] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.880103] iommu: Default domain type: Passthrough [ 0.883425] SCSI subsystem initialized [ 0.885121] ACPI: bus type USB registered [ 0.887194] usbcore: registered new interface driver usbfs [ 0.888125] usbcore: registered new interface driver hub [ 0.891085] usbcore: registered new device driver usb [ 0.892213] pps_core: LinuxPPS API ver. 1 registered [ 0.895070] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.898062] PTP clock support registered [ 0.900115] EDAC MC: Ver: 3.0.0 [ 0.903138] PCI: Using ACPI for IRQ routing [ 0.906107] NetLabel: Initializing [ 0.910014] NetLabel: domain hash size = 128 [ 0.916031] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.922084] NetLabel: unlabeled traffic allowed by default [ 0.927049] vgaarb: loaded [ 0.930085] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.934206] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.943203] clocksource: Switched to clocksource kvm-clock [ 1.066132] VFS: Disk quotas dquot_6.6.0 [ 1.067791] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.070153] *** VALIDATE ramfs *** [ 1.071061] *** VALIDATE hugetlbfs *** [ 1.072704] pnp: PnP ACPI init [ 1.075212] pnp: PnP ACPI: found 6 devices [ 1.092813] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.095827] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.098163] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.100551] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.103046] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.105473] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.108473] NET: Registered protocol family 2 [ 1.111098] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.116718] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.120989] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.126593] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.129925] TCP: Hash tables configured (established 65536 bind 65536) [ 1.132758] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.135642] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.138455] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.140994] NET: Registered protocol family 1 [ 1.143305] RPC: Registered named UNIX socket transport module. [ 1.144843] RPC: Registered udp transport module. [ 1.145931] RPC: Registered tcp transport module. [ 1.147430] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.149371] NET: Registered protocol family 44 [ 1.150932] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.153164] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.155287] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.157477] PCI: CLS 0 bytes, default 64 [ 1.159870] Unpacking initramfs... [ 2.863433] debug: unmapping init [mem 0xffff91b0bcc54000-0xffff91b0bffbffff] [ 2.867304] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.869445] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.872749] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.414371] Initialise system trusted keyrings [ 3.415726] Key type blacklist registered [ 3.417125] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.425104] zbud: loaded [ 3.427759] *** VALIDATE nfs *** [ 3.428776] *** VALIDATE nfs4 *** [ 3.430019] pstore: using deflate compression [ 3.433267] Platform Keyring initialized [ 3.532535] NET: Registered protocol family 38 [ 3.533914] Key type asymmetric registered [ 3.535109] Asymmetric key parser 'x509' registered [ 3.536730] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.539352] io scheduler mq-deadline registered [ 3.540669] io scheduler kyber registered [ 3.541772] io scheduler bfq registered [ 3.543449] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.547530] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.550620] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.552967] ACPI: Power Button [PWRF] [ 3.558282] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.564468] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.582709] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.592643] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.611699] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.642713] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.672981] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.678402] Non-volatile memory driver v1.3 [ 3.681213] Linux agpgart interface v0.103 [ 3.724623] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.728970] vda: detected capacity change from 0 to 69001216 [ 3.745458] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.747962] vdb: detected capacity change from 0 to 1073741824 [ 3.986640] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.991792] vdc: detected capacity change from 0 to 2621440000 [ 4.128404] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.131868] vdd: detected capacity change from 0 to 2621440000 [ 4.216612] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.220910] vde: detected capacity change from 0 to 4294967296 [ 4.377426] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.381964] vdf: detected capacity change from 0 to 4294967296 [ 4.390035] libphy: Fixed MDIO Bus: probed [ 4.400570] usbcore: registered new interface driver usbserial_generic [ 4.403844] usbserial: USB Serial support registered for generic [ 4.406955] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.412927] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.415609] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.418693] mousedev: PS/2 mouse device common for all mice [ 4.421613] rtc_cmos 00:05: RTC can wake from S4 [ 4.424602] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.424808] rtc_cmos 00:05: registered as rtc0 [ 4.432219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.433149] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.444987] intel_pstate: CPU model not supported [ 4.448300] hid: raw HID events driver (C) Jiri Kosina [ 4.450345] usbcore: registered new interface driver usbhid [ 4.452642] usbhid: USB HID core driver [ 4.453923] drop_monitor: Initializing network drop monitor service [ 4.456887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.457559] Initializing XFRM netlink socket [ 4.463873] NET: Registered protocol family 10 [ 4.467980] Segment Routing with IPv6 [ 4.469628] NET: Registered protocol family 17 [ 4.472525] mpls_gso: MPLS GSO support [ 4.479589] RAS: Correctable Errors collector initialized. [ 4.482411] AVX version of gcm_enc/dec engaged. [ 4.484071] AES CTR mode by8 optimization enabled [ 4.567588] sched_clock: Marking stable (4567521066, 0)->(5722390311, -1154869245) [ 4.571423] registered taskstats version 1 [ 4.573837] Loading compiled-in X.509 certificates [ 4.576164] zswap: loaded using pool lzo/zbud [ 4.608502] Key type big_key registered [ 4.622075] Key type encrypted registered [ 4.624450] ima: No TPM chip found, activating TPM-bypass! [ 4.627491] ima: Allocated hash algorithm: sha1 [ 4.629975] ima: No architecture policies found [ 4.633127] evm: Initialising EVM extended attributes: [ 4.636087] evm: security.selinux [ 4.637776] evm: security.ima [ 4.639474] evm: security.capability [ 4.641480] evm: HMAC attrs: 0x1 [ 4.644569] rtc_cmos 00:05: setting system clock to 2026-05-27 08:13:19 UTC (1779869599) [ 4.653099] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 4.655928] debug: unmapping init [mem 0xffffffff8d782000-0xffffffff8da58fff] [ 4.660111] Write protecting the kernel read-only data: 28672k [ 4.664278] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 4.667278] debug: unmapping init [mem 0xffffffff8c714000-0xffffffff8c7fffff] [ 4.759198] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.773546] systemd[1]: Detected virtualization kvm. [ 4.775851] systemd[1]: Detected architecture x86-64. [ 4.778376] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.809111] systemd[1]: No hostname configured. [ 4.811163] systemd[1]: Set hostname to . [ 4.813332] random: systemd: uninitialized urandom read (16 bytes read) [ 4.815854] systemd[1]: Initializing machine ID from random generator. [ 4.909118] random: ln: uninitialized urandom read (6 bytes read) [ 5.071925] random: systemd: uninitialized urandom read (16 bytes read) [ 5.079675] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.087539] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.099994] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.802989] device-mapper: uevent: version 1.0.3 [ 7.806221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.304367] virtio_net virtio0 ens2: renamed from eth0 [ 9.308878] random: fast init done [ 9.428067] scsi host0: ata_piix [ 9.439396] scsi host1: ata_piix [ 9.443319] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.451945] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.149277] random: crng init done [ 14.157568] random: 7 urandom warning(s) missed due to ratelimiting [ 14.186898] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.763179] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.667582] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.431838] SELinux: Disabled at runtime. [ 21.505251] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.517267] systemd[1]: Detected virtualization kvm. [ 21.519547] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.322858] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.350942] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.017601] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.025495] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.030845] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.154275] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.185187] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 25.133245] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started Flush Journal to Persistent Storage. [ 26.476182] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.064469] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.382941] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.417411] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.509339] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 35.783718] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (12s / no limit)[ 36.130948] NFS: Registering the id_resolver key type [ 36.133408] Key type id_resolver registered [ 36.135386] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-server login: [ 76.862084] hrtimer: interrupt took 2122080 ns [ 93.036411] libcfs: loading out-of-tree module taints kernel. [ 93.097598] Key type ._llcrypt registered [ 93.103233] Key type .llcrypt registered [ 93.251598] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 113.135934] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 114.797313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.807733] alg: No test for adler32 (adler32-zlib) [ 116.305660] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 117.166162] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 118.975229] Key type lgssc registered [ 120.987694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.830573] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.054292] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 189.823471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.882187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.133454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.157804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.225444] Lustre: lustre-MDT0000: new disk, initializing [ 191.299334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.315717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 195.286778] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.263614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.409878] Lustre: 6508:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.487355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 208.496807] Lustre: Skipped 1 previous similar message [ 208.647341] Lustre: lustre-MDT0001: new disk, initializing [ 208.769429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.802239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 208.815123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.736349] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.833570] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 231.411618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 231.702411] Lustre: lustre-OST0000: new disk, initializing [ 231.706609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 231.717519] Lustre: 8412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 231.802161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.859855] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.647767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.659617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.722808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.543178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.636757] Lustre: lustre-OST0001: new disk, initializing [ 252.640401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.646435] Lustre: 9469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.694814] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.918390] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.628276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.639958] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.691980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.536397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.378873] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.709870] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing check_logdir /tmp/testlogs/ [ 291.191056] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing yml_node [ 296.224897] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 299.399434] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 301.935149] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 303.814436] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed May 27 04:18:16 EDT 2026 [ 320.098708] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 321.567798] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 341.477177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.491178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 341.503407] Lustre: Skipped 3 previous similar messages [ 341.519281] Lustre: Skipped 3 previous similar messages [ 346.595582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.600218] Lustre: Skipped 1 previous similar message [ 351.199232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 351.436899] Lustre: server umount lustre-MDT0000 complete [ 355.175328] LustreError: 9468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779869950 with bad export cookie 1747131638662540047 [ 355.181530] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.190632] LustreError: 9468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 356.832294] LustreError: 8854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.836276] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.846985] LustreError: 8854:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 356.881587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.888831] Lustre: Skipped 1 previous similar message [ 360.864029] LustreError: 8854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.864732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 360.893752] Lustre: Skipped 1 previous similar message [ 361.807990] Lustre: server umount lustre-MDT0001 complete [ 365.934803] Lustre: server umount lustre-OST0000 complete [ 370.138857] Lustre: server umount lustre-OST0001 complete [ 375.967667] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 384.741275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 418.463801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.670616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.701841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.770625] Lustre: lustre-MDT0000: new disk, initializing [ 418.882801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.911547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 422.593577] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 434.155991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.313373] Lustre: 14980:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.382738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.391623] Lustre: Skipped 1 previous similar message [ 434.448902] Lustre: lustre-MDT0001: new disk, initializing [ 434.538979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.592171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.605196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.584823] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 446.660463] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.657616] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 457.311431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 457.578935] Lustre: lustre-OST0000: new disk, initializing [ 457.588412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 457.597511] Lustre: 16724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 457.692346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 459.437561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 459.457035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 459.665890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 464.051449] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 470.929776] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.155492] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 475.370398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.002525] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.219344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 485.354294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 485.360049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.361927] LustreError: Skipped 1 previous similar message [ 485.375950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 485.380042] Lustre: Skipped 2 previous similar messages [ 485.389915] Lustre: Skipped 1 previous similar message [ 488.314736] Lustre: server umount lustre-OST0000 complete [ 495.584570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.592314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 495.603535] Lustre: Skipped 1 previous similar message [ 498.007025] Lustre: server umount lustre-MDT0000 complete [ 500.707407] LustreError: 15870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 500.724369] LustreError: 15870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 501.923989] LustreError: 16734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870096 with bad export cookie 1747131638662541580 [ 501.938699] LustreError: 16734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 501.950414] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.386444] Lustre: server umount lustre-MDT0001 complete [ 505.635135] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:21:38 (1779870098) [ 511.652967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.301772] LustreError: 18842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.397640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.089344] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 517.604681] LustreError: 18843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.411965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.918484] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 534.926320] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 538.063797] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 544.854201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.141830] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 555.755462] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 567.777143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.778425] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 567.792500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 567.792509] Lustre: Skipped 1 previous similar message [ 567.796618] Lustre: Skipped 1 previous similar message [ 573.278426] Lustre: server umount lustre-OST0000 complete [ 578.020467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.050122] Lustre: Skipped 1 previous similar message [ 583.405168] Lustre: server umount lustre-MDT0000 complete [ 586.573752] LustreError: 18823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870181 with bad export cookie 1747131638662542462 [ 586.585512] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.587816] LustreError: 18823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 586.831147] Lustre: server umount lustre-MDT0001 complete [ 595.526661] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 597.716243] Key type lgssc unregistered [ 597.963409] LNet: 22017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.968569] LNetError: 22017:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.999600] LNet: Removed LNI 192.168.201.141@tcp [ 598.876208] Key type .llcrypt unregistered [ 598.877741] Key type ._llcrypt unregistered [ 616.874382] Key type ._llcrypt registered [ 616.875881] Key type .llcrypt registered [ 616.958338] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 617.654670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 617.686101] alg: No test for adler32 (adler32-zlib) [ 618.753555] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 618.986370] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 620.607174] Key type lgssc registered [ 621.527713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 627.444377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 627.473231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.905455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.840295] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 638.127260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.435338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.873994] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 646.631270] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 649.643593] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 654.872191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 655.144735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.249277] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 662.900662] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 674.618551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 678.880712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 678.886274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.894983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 679.909442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.910019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 679.912534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 680.514536] Lustre: server umount lustre-OST0000 complete [ 684.513815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 684.517817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.530812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.535028] Lustre: Skipped 1 previous similar message [ 690.145538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 690.155233] Lustre: Skipped 3 previous similar messages [ 695.264446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.267675] Lustre: Skipped 1 previous similar message [ 697.311189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 697.462913] Lustre: server umount lustre-MDT0000 complete [ 700.245843] LustreError: 26277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870295 with bad export cookie 306074140903793476 [ 700.252317] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.257184] LustreError: 26277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 700.475943] Lustre: server umount lustre-MDT0001 complete [ 708.923973] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 711.104511] Key type lgssc unregistered [ 711.381643] LNet: 27012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.390200] LNetError: 27012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.403085] LNet: Removed LNI 192.168.201.141@tcp [ 712.051217] Key type .llcrypt unregistered [ 712.052808] Key type ._llcrypt unregistered [ 727.066373] Key type ._llcrypt registered [ 727.067793] Key type .llcrypt registered [ 727.127717] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 727.700664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 727.759588] alg: No test for adler32 (adler32-zlib) [ 728.694624] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 728.805597] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 730.431157] Key type lgssc registered [ 731.130589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 735.653174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 735.669412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.924762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.398778] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 744.830480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.155657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 748.308693] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 752.531608] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 754.993931] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 764.176720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 764.598422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.921856] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 773.267326] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 782.000276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 790.497788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 790.500856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.516833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.544711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.557538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.558353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 794.378565] Lustre: server umount lustre-OST0000 complete [ 797.668942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.676139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 797.686373] Lustre: Skipped 1 previous similar message [ 797.690145] Lustre: Skipped 1 previous similar message [ 802.784695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.791947] Lustre: Skipped 2 previous similar messages [ 803.137842] Lustre: server umount lustre-MDT0000 complete [ 805.747725] LustreError: 28365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870400 with bad export cookie 16329515239182101924 [ 805.749359] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 805.758374] LustreError: 28365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 805.972335] Lustre: server umount lustre-MDT0001 complete [ 813.208566] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 814.991178] Key type lgssc unregistered [ 815.189205] LNet: 31924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.198123] LNetError: 31924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.216161] LNet: Removed LNI 192.168.201.141@tcp [ 815.693361] Key type .llcrypt unregistered [ 815.695277] Key type ._llcrypt unregistered [ 830.106661] Key type ._llcrypt registered [ 830.108389] Key type .llcrypt registered [ 830.170818] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 830.823819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.833050] alg: No test for adler32 (adler32-zlib) [ 831.812440] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 831.968145] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 833.599209] Key type lgssc registered [ 834.190713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 838.639942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 838.660603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.975680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.996831] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 848.849445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.102629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 851.708629] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 855.499734] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.769326] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 861.549609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.763325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.338285] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 865.904534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 868.434481] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.586226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.586258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 879.603597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 879.618740] Lustre: Skipped 1 previous similar message [ 880.439670] Lustre: server umount lustre-OST0000 complete [ 884.707221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.711905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.718217] Lustre: Skipped 2 previous similar messages [ 889.076621] Lustre: server umount lustre-MDT0000 complete [ 889.824135] LustreError: 33298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.839818] LustreError: 33298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 892.186714] LustreError: 33277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870487 with bad export cookie 15863214103774886340 [ 892.202364] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 892.462235] Lustre: server umount lustre-MDT0001 complete [ 899.751588] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 901.585122] Key type lgssc unregistered [ 901.801279] LNet: 36471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.811476] LNetError: 36471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.823707] LNet: Removed LNI 192.168.201.141@tcp [ 903.475704] Key type .llcrypt unregistered [ 903.477325] Key type ._llcrypt unregistered [ 919.221263] Key type ._llcrypt registered [ 919.223743] Key type .llcrypt registered [ 919.303725] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 920.163313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.187072] alg: No test for adler32 (adler32-zlib) [ 921.131950] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 921.279409] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 922.911753] Key type lgssc registered [ 923.620376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 928.861117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 928.874607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.130064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.317892] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 938.670681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.887098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 942.109184] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 946.332722] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.889557] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 953.060692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 953.286185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 954.288660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 957.206657] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 960.569666] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 969.697118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.701063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.709121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 973.169525] Lustre: lustre-OST0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 973.798627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 973.806373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.510747] Lustre: server umount lustre-OST0000 complete [ 993.187205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.192230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.196719] Lustre: Skipped 1 previous similar message [ 993.208297] Lustre: Skipped 3 previous similar messages [ 998.369352] LustreError: 37849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.385107] LustreError: 37849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 998.425749] Lustre: server umount lustre-MDT0000 complete [ 1000.754822] LustreError: 37829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870595 with bad export cookie 6518611994672142326 [ 1000.755671] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.760197] LustreError: 37829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1015.263122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1015.419418] Lustre: server umount lustre-MDT0001 complete [ 1023.277484] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1025.150519] Key type lgssc unregistered [ 1025.444871] LNet: 41200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.449606] LNetError: 41200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.470331] LNet: Removed LNI 192.168.201.141@tcp [ 1027.100352] Key type .llcrypt unregistered [ 1027.101567] Key type ._llcrypt unregistered [ 1041.428157] Key type ._llcrypt registered [ 1041.430117] Key type .llcrypt registered [ 1041.500777] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1041.992270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1042.004747] alg: No test for adler32 (adler32-zlib) [ 1042.919675] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1043.033936] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1044.647131] Key type lgssc registered [ 1045.384325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1050.380188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.399807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.735583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.794154] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1060.922306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.213349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.995476] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1067.717402] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.011101] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1074.623373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1074.874491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.935097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1079.227508] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1082.621552] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1092.066093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1092.072654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.083855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.579132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.585209] Lustre: Skipped 1 previous similar message [ 1092.588441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.591970] Lustre: Skipped 1 previous similar message [ 1094.521407] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 1096.913869] Lustre: server umount lustre-MDT0000 complete [ 1097.697382] LustreError: 42570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1097.708645] LustreError: 42570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1098.935985] LustreError: 42555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870693 with bad export cookie 15227774648549514844 [ 1098.938984] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.945584] LustreError: 42555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1099.613226] LustreError: 43701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.627220] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 1102.818021] LustreError: 42569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.819030] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.840746] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1103.843865] Lustre: Skipped 1 previous similar message [ 1105.174594] Lustre: server umount lustre-MDT0001 complete [ 1131.999131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1132.129583] Lustre: server umount lustre-OST0000 complete [ 1140.356315] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1141.766986] Key type lgssc unregistered [ 1141.911793] LNet: 46073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.916521] LNetError: 46073:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.925507] LNet: Removed LNI 192.168.201.141@tcp [ 1142.307215] Key type .llcrypt unregistered [ 1142.309014] Key type ._llcrypt unregistered [ 1154.610908] Key type ._llcrypt registered [ 1154.612290] Key type .llcrypt registered [ 1154.674966] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1155.215645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1155.241952] alg: No test for adler32 (adler32-zlib) [ 1156.171323] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1156.295113] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1157.911278] Key type lgssc registered [ 1158.402974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1161.997356] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.004949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.439577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1182.559514] LustreError: 47421:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1209.183242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.709101] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1214.088632] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1378.059421] Lustre: server umount lustre-OST0000 complete [ 1386.470498] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1387.797497] Key type lgssc unregistered [ 1387.966635] LNet: 48963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.970231] LNetError: 48963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.981531] LNet: Removed LNI 192.168.201.141@tcp [ 1388.376995] Key type .llcrypt unregistered [ 1388.380206] Key type ._llcrypt unregistered [ 1400.717423] Key type ._llcrypt registered [ 1400.718911] Key type .llcrypt registered [ 1400.789852] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1401.352495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.387278] alg: No test for adler32 (adler32-zlib) [ 1402.299836] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1402.430398] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1404.047164] Key type lgssc registered [ 1404.630541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.094555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1409.105401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.384475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1412.980947] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1418.939576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1419.178256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1421.815571] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1425.392819] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1427.767408] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1432.152735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.387123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.498157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1436.405131] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1439.861769] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1443.366370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.371877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.499481] Lustre: server umount lustre-OST0000 complete [ 1453.539804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1453.542044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1453.549869] Lustre: Skipped 1 previous similar message [ 1453.554504] Lustre: Skipped 1 previous similar message [ 1456.254904] Lustre: server umount lustre-MDT0000 complete [ 1458.659758] LustreError: 50338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1458.680275] LustreError: 50338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1458.955849] LustreError: 50319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871053 with bad export cookie 2127014367246694325 [ 1458.957497] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1458.964535] LustreError: 50319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1459.220402] Lustre: server umount lustre-MDT0001 complete [ 1465.613197] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1467.196132] Key type lgssc unregistered [ 1467.410946] LNet: 53508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.416320] LNetError: 53508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.426878] LNet: Removed LNI 192.168.201.141@tcp [ 1467.862893] Key type .llcrypt unregistered [ 1467.868584] Key type ._llcrypt unregistered [ 1482.178800] Key type ._llcrypt registered [ 1482.180189] Key type .llcrypt registered [ 1482.235641] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1482.809106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1482.891748] alg: No test for adler32 (adler32-zlib) [ 1483.861875] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1484.038255] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1485.679199] Key type lgssc registered [ 1486.383624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1491.225547] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1491.244937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1506.719359] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1511.839205] LustreError: 54856:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1538.463172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.222651] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1545.614895] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1550.004819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1550.325650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.368623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1552.584507] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1557.125928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.302945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1559.122486] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1561.848705] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1563.963149] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1565.664961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1565.667527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.674170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.717613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.718328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1567.728757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.153951] Lustre: server umount lustre-OST0000 complete [ 1592.290542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.294992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1592.299642] Lustre: Skipped 1 previous similar message [ 1592.305158] Lustre: Skipped 2 previous similar messages [ 1594.762025] Lustre: server umount lustre-MDT0000 complete [ 1597.387269] LustreError: 54862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871192 with bad export cookie 225875593706116090 [ 1597.397626] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1597.402869] LustreError: 54862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1603.690906] Lustre: server umount lustre-MDT0001 complete [ 1608.616956] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1610.065110] Key type lgssc unregistered [ 1610.256518] LNet: 58239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.261773] LNetError: 58239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.319284] LNet: Removed LNI 192.168.201.141@tcp [ 1611.810422] Key type .llcrypt unregistered [ 1611.813726] Key type ._llcrypt unregistered [ 1624.499172] Key type ._llcrypt registered [ 1624.501119] Key type .llcrypt registered [ 1624.579275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1625.215086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1625.237361] alg: No test for adler32 (adler32-zlib) [ 1626.120931] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1626.232629] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1627.831224] Key type lgssc registered [ 1628.440502] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1632.391672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1632.407269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.693400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.764865] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1639.991744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1640.185895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1642.090852] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1644.717295] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1646.668152] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1650.057543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.219667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1653.125684] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1655.996119] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1658.414959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1671.648887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1671.655501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.667370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.700015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.705136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.709061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1673.722305] Lustre: Skipped 1 previous similar message [ 1676.316828] Lustre: server umount lustre-OST0000 complete [ 1678.817646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.820868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.826398] Lustre: Skipped 1 previous similar message [ 1683.936031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.938223] Lustre: Skipped 2 previous similar messages [ 1684.164437] Lustre: server umount lustre-MDT0000 complete [ 1686.174642] LustreError: 59595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871281 with bad export cookie 17905470182441922857 [ 1686.179271] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.183202] LustreError: 59595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1686.337897] Lustre: server umount lustre-MDT0001 complete [ 1692.751865] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1694.221802] Key type lgssc unregistered [ 1694.400541] LNet: 62809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.405164] LNetError: 62809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.420667] LNet: Removed LNI 192.168.201.141@tcp [ 1694.758756] Key type .llcrypt unregistered [ 1694.760089] Key type ._llcrypt unregistered [ 1707.076229] Key type ._llcrypt registered [ 1707.077652] Key type .llcrypt registered [ 1707.122663] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1707.452648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1707.460364] alg: No test for adler32 (adler32-zlib) [ 1708.363595] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1708.474520] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1710.087180] Key type lgssc registered [ 1710.626846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.046216] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:41:49 (1779871309) [ 1718.958131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.980677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.274388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.942763] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1726.859268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.048904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1729.102401] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1731.423894] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1732.960082] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1735.894294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.029235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.765174] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1741.498882] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1746.286738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1750.737447] Lustre: Failing over lustre-MDT0000 [ 1750.875644] Lustre: server umount lustre-MDT0000 complete [ 1751.522190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.522980] LustreError: 64474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1751.531791] Lustre: Skipped 1 previous similar message [ 1751.547717] LustreError: 64474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1753.446882] LustreError: 66621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1755.322688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.378937] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1755.442413] Lustre: *** cfs_fail_loc=135, val=0*** [ 1755.485919] LustreError: 67221:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 1755.489144] LustreError: 67221:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -2 [ 1755.492676] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1755.492676] [ 1755.496041] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1755.502804] LustreError: 67209:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1755.505468] LustreError: 67209:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 1755.509313] LustreError: 67209:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1755.511564] LustreError: 67209:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1755.523267] Lustre: server umount lustre-MDT0000 complete [ 1755.525817] LustreError: 67209:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1756.642602] LustreError: 64475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.651838] LustreError: 64475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1758.515222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.602542] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.736335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.775729] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1760.705474] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1763.675399] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1763.819051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1763.831852] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1763.870779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1767.904629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1767.907585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.912246] Lustre: Skipped 1 previous similar message [ 1767.914573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1768.936503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1768.938596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1768.949126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1771.295042] Lustre: server umount lustre-OST0000 complete [ 1774.048399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1774.051898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1774.070834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1774.073286] Lustre: Skipped 2 previous similar messages [ 1779.171781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1779.177301] Lustre: Skipped 1 previous similar message [ 1784.291584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1784.296185] Lustre: Skipped 1 previous similar message [ 1787.359125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1787.435454] Lustre: server umount lustre-MDT0000 complete [ 1789.111411] LustreError: 66712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871383 with bad export cookie 11971430736375579312 [ 1789.111866] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.118535] LustreError: 66712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1789.295316] Lustre: server umount lustre-MDT0001 complete [ 1794.848185] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1796.228217] Key type lgssc unregistered [ 1796.411529] LNet: 68959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.416996] LNetError: 68959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.435836] LNet: Removed LNI 192.168.201.141@tcp [ 1796.875383] Key type .llcrypt unregistered [ 1796.877206] Key type ._llcrypt unregistered [ 1811.899497] Key type ._llcrypt registered [ 1811.901247] Key type .llcrypt registered [ 1811.995717] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1813.084103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1813.113605] alg: No test for adler32 (adler32-zlib) [ 1814.134996] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1814.362165] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1816.015574] Key type lgssc registered [ 1816.939932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.916307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.977237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.049887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.950476] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1836.582633] LustreError: 70334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.610962] LustreError: 70334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1841.638162] LustreError: 70333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.568936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.173377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1850.746289] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1860.273507] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1866.737769] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.080153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.120606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.259197] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1896.538424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1897.096506] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1905.376858] Lustre: Failing over lustre-MDT0000 [ 1905.645400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.665868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.698120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1906.207776] Lustre: server umount lustre-MDT0000 complete [ 1906.658242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.659515] LustreError: 70334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.744518] LustreError: 70334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1910.206926] LustreError: 70330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.233550] LustreError: 70329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.254815] LustreError: 70329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1918.677511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.142060] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.403572] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1919.649639] LustreError: 73085:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 1919.656962] LustreError: 73085:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -9 [ 1919.669950] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1919.669950] [ 1919.686837] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1919.699821] LustreError: 73073:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1919.711662] LustreError: 73073:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -9 [ 1919.738089] LustreError: 73073:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 1919.747333] LustreError: 73073:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 1919.794879] Lustre: server umount lustre-MDT0000 complete [ 1919.798161] LustreError: 73073:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1925.495445] LustreError: 70330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.511348] LustreError: 70330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 1927.387946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1927.476298] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.739438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.802749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1930.595813] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1932.798751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1932.853092] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1932.861563] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1932.913558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1943.015529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1943.035553] Lustre: Skipped 1 previous similar message [ 1943.048111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.966817] Lustre: server umount lustre-OST0000 complete [ 1953.250438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.256125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.275400] Lustre: Skipped 1 previous similar message [ 1953.287737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.312789] Lustre: Skipped 2 previous similar messages [ 1956.773987] Lustre: server umount lustre-MDT0000 complete [ 1958.368507] LustreError: 71044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.417041] LustreError: 71044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1960.521893] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871555 with bad export cookie 2984925641117337197 [ 1960.523983] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1960.528623] LustreError: 70312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1960.856635] Lustre: server umount lustre-MDT0001 complete [ 1971.141983] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1973.491591] Key type lgssc unregistered [ 1973.723870] LNet: 74830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.733431] LNetError: 74830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.762818] LNet: Removed LNI 192.168.201.141@tcp [ 1974.523184] Key type .llcrypt unregistered [ 1974.526117] Key type ._llcrypt unregistered [ 1986.578142] Key type ._llcrypt registered [ 1986.580879] Key type .llcrypt registered [ 1986.664033] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1986.934383] Key type .llcrypt unregistered [ 1986.939740] Key type ._llcrypt unregistered [ 2015.447297] Key type ._llcrypt registered [ 2015.456172] Key type .llcrypt registered [ 2015.658448] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2017.478182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2017.504447] alg: No test for adler32 (adler32-zlib) [ 2018.883880] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2019.186049] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2020.983992] Key type lgssc registered [ 2022.281624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2030.859976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2030.881510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.281032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.274724] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2042.862897] LustreError: 76696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2042.897270] LustreError: 76696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2047.841212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.978710] LustreError: 76697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.352562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2053.171607] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2059.109257] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2063.471757] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2071.295730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.605241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.713929] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2077.830558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2083.584965] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2093.537794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.544530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.566952] Lustre: Skipped 1 previous similar message [ 2093.588977] Lustre: Skipped 1 previous similar message [ 2098.659027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2098.671236] Lustre: Skipped 1 previous similar message [ 2099.858779] Lustre: server umount lustre-OST0000 complete [ 2103.777968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.787484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.787703] Lustre: Skipped 1 previous similar message [ 2103.798180] Lustre: Skipped 1 previous similar message [ 2108.897386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.902678] Lustre: Skipped 1 previous similar message [ 2114.016916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.022924] Lustre: Skipped 1 previous similar message [ 2118.111160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2118.313879] Lustre: server umount lustre-MDT0000 complete [ 2119.136919] LustreError: 77955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.174524] LustreError: 77955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2122.139504] LustreError: 79139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871716 with bad export cookie 2856253041456572992 [ 2122.147792] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.154025] LustreError: 79139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2122.532212] Lustre: server umount lustre-MDT0001 complete [ 2136.060553] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2140.685232] Key type lgssc unregistered [ 2141.232676] LNet: 79876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.248965] LNetError: 79876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.297673] LNet: Removed LNI 192.168.201.141@tcp [ 2142.580267] Key type .llcrypt unregistered [ 2142.581961] Key type ._llcrypt unregistered [ 2168.043375] Key type ._llcrypt registered [ 2168.045254] Key type .llcrypt registered [ 2168.177937] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2169.323567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.476197] alg: No test for adler32 (adler32-zlib) [ 2170.639449] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2170.902653] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2172.535487] Key type lgssc registered [ 2173.579471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2180.825559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.845210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.270433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.382544] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2192.864631] LustreError: 81272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.894694] LustreError: 81272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2196.917199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.414984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2203.261879] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2209.438820] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.754991] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2220.593550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2220.946586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.691270] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2228.161184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2232.832500] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2240.989198] Lustre: server umount lustre-OST0000 complete [ 2248.676756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.686444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2248.707445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.795177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.797642] Lustre: Skipped 2 previous similar messages [ 2254.891445] Lustre: server umount lustre-MDT0000 complete [ 2258.929927] LustreError: 81272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.956544] LustreError: 81272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2259.807941] Lustre: 80584:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779871838/real 1779871838] req@00000000b61b98bd x1866330815881728/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779871854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2259.851870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2259.879261] Lustre: Skipped 1 previous similar message [ 2260.457952] LustreError: 81253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871855 with bad export cookie 10629135228937581859 [ 2260.462980] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.478946] LustreError: 81253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2260.966243] Lustre: server umount lustre-MDT0001 complete [ 2273.476891] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2276.757912] Key type lgssc unregistered [ 2277.223933] LNet: 84446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.241264] LNetError: 84446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2277.262378] LNet: Removed LNI 192.168.201.141@tcp [ 2278.469923] Key type .llcrypt unregistered [ 2278.472171] Key type ._llcrypt unregistered [ 2304.470557] Key type ._llcrypt registered [ 2304.475206] Key type .llcrypt registered [ 2304.606923] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2306.262059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2306.281713] alg: No test for adler32 (adler32-zlib) [ 2307.376493] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2307.597681] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2309.327518] Key type lgssc registered [ 2310.514429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2318.688849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2318.705736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.282728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2325.266155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2330.594033] LustreError: 85862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2330.622845] LustreError: 85862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2335.722847] LustreError: 85863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.812310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.354604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2341.380633] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2347.853562] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2351.887498] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2358.848712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2359.536458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.707091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2367.063370] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2373.850160] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2385.893205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2385.904486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.942919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.412893] Lustre: server umount lustre-OST0000 complete [ 2396.643992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.658534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.675474] Lustre: Skipped 2 previous similar messages [ 2396.682034] Lustre: Skipped 3 previous similar messages [ 2401.761168] LustreError: 87371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.926912] Lustre: server umount lustre-MDT0000 complete [ 2406.880892] LustreError: 85857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.905928] LustreError: 85857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2406.924546] LustreError: 85842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872001 with bad export cookie 11367049491453764362 [ 2406.939083] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.303391] Lustre: server umount lustre-MDT0001 complete [ 2420.073504] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2423.940531] Key type lgssc unregistered [ 2424.214814] LNet: 89040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.220210] LNetError: 89040:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.263866] LNet: Removed LNI 192.168.201.141@tcp [ 2426.333375] Key type .llcrypt unregistered [ 2426.335330] Key type ._llcrypt unregistered [ 2453.838766] Key type ._llcrypt registered [ 2453.840453] Key type .llcrypt registered [ 2453.983355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2455.371716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.625580] alg: No test for adler32 (adler32-zlib) [ 2456.786314] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2457.083728] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2458.783182] Key type lgssc registered [ 2460.305377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2469.650868] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2469.679437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2485.407350] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2490.527429] LustreError: 90414:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 2517.215305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.411158] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2530.106188] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2537.253319] Lustre: server umount lustre-OST0000 complete [ 2547.188127] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:55:40 (1779872140) [ 2553.981568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.561126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2559.968880] LustreError: 91940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.999614] LustreError: 91940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2560.011953] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2565.091347] LustreError: 91939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.208850] LustreError: 91940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.903023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.382936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2577.408843] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2585.558743] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2590.438714] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2598.009694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.433921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.489721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2606.290516] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2612.998459] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2627.044603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.046345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.068668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.085697] Lustre: Skipped 1 previous similar message [ 2627.107404] Lustre: Skipped 1 previous similar message [ 2632.165869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.173451] Lustre: Skipped 1 previous similar message [ 2632.547472] Lustre: server umount lustre-OST0000 complete [ 2637.282075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.307187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.400280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.406752] Lustre: Skipped 2 previous similar messages [ 2642.710635] Lustre: server umount lustre-MDT0000 complete [ 2646.051967] LustreError: 91921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872240 with bad export cookie 3804974700779725359 [ 2646.060842] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.067028] LustreError: 91921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2646.354670] Lustre: server umount lustre-MDT0001 complete [ 2657.269142] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2660.209816] Key type lgssc unregistered [ 2660.623158] LNet: 95276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.633098] LNetError: 95276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.653222] LNet: Removed LNI 192.168.201.141@tcp [ 2661.419091] Key type .llcrypt unregistered [ 2661.423926] Key type ._llcrypt unregistered [ 2706.472596] Key type ._llcrypt registered [ 2706.474245] Key type .llcrypt registered [ 2706.575786] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2707.552937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2707.594915] alg: No test for adler32 (adler32-zlib) [ 2708.614804] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2708.897868] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2710.567170] Key type lgssc registered [ 2711.673957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2718.883621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2718.902919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.421674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.443635] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2730.986847] LustreError: 97446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.010207] LustreError: 97446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2732.612959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.038722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2737.050676] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2742.551606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2746.253692] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2753.187426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2753.530564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.865109] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2765.814366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 2766.112165] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2776.040192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.046991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2776.071320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2776.084613] Lustre: Skipped 1 previous similar message [ 2780.737804] Lustre: server umount lustre-OST0000 complete [ 2786.274977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.276299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.292477] Lustre: Skipped 2 previous similar messages [ 2786.312944] Lustre: Skipped 1 previous similar message [ 2790.816920] Lustre: server umount lustre-MDT0000 complete [ 2791.399627] LustreError: 97443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2791.428834] LustreError: 97443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2795.101683] LustreError: 97426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872389 with bad export cookie 18194025523703020917 [ 2795.103103] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2795.109791] LustreError: 97426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2795.399323] Lustre: server umount lustre-MDT0001 complete [ 2806.105627] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2808.877817] Key type lgssc unregistered [ 2809.195782] LNet: 100621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2809.200303] LNetError: 100621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2809.222048] LNet: Removed LNI 192.168.201.141@tcp [ 2810.253897] Key type .llcrypt unregistered [ 2810.255489] Key type ._llcrypt unregistered [ 2828.809486] Key type ._llcrypt registered [ 2828.811466] Key type .llcrypt registered [ 2828.907912] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2830.166360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.299721] alg: No test for adler32 (adler32-zlib) [ 2831.370462] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 2831.635665] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2833.384712] Key type lgssc registered [ 2835.109821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2843.365817] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2843.380970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.911349] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2864.031296] LustreError: 101845:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 2890.720295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.516437] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2902.656401] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.785038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.987128] LustreError: 101182:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003704ed26 x1866331508967680/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2911.178062] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2911.191767] LustreError: 102770:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2911.203769] LustreError: 102770:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 2911.211293] LustreError: 102770:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 2911.321473] Lustre: server umount lustre-MDT0000 complete [ 2911.325118] LustreError: 102770:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2932.703158] Lustre: 101183:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779872511/real 1779872511] req@00000000f64b5981 x1866331508968832/t0(0) o400->MGC192.168.201.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779872527 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2932.734768] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2932.756947] LustreError: 101857:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2932.954196] Lustre: server umount lustre-OST0000 complete [ 2943.117922] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2953.943361] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2993.858687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.310412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2994.348663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2994.440708] Lustre: lustre-MDT0000: new disk, initializing [ 2994.551280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.590476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3000.373666] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3013.173380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3013.348375] Lustre: 105527:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3013.430373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3013.433923] Lustre: Skipped 1 previous similar message [ 3013.560217] Lustre: lustre-MDT0001: new disk, initializing [ 3013.667371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3013.702940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3013.708304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3018.352804] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3025.360513] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3028.533239] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3034.991492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.426584] Lustre: lustre-OST0000: new disk, initializing [ 3035.435571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3035.447236] Lustre: 107270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3035.549838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.863298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3041.889694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3041.987688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3041.996314] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3050.196669] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3055.569893] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3055.759386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3060.827479] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3061.057983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3067.879773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.885213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.904575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.943881] Lustre: Skipped 1 previous similar message [ 3069.495562] Lustre: server umount lustre-OST0000 complete [ 3075.050138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3075.063387] LustreError: Skipped 1 previous similar message [ 3075.065178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.091881] Lustre: Skipped 1 previous similar message [ 3075.099736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.113891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.123357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.046389] Lustre: server umount lustre-MDT0000 complete [ 3084.738204] LustreError: 105519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872679 with bad export cookie 9021148374627937317 [ 3084.740716] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.749526] LustreError: 105519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3085.156929] Lustre: server umount lustre-MDT0001 complete [ 3094.838602] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:04:48 (1779872688) [ 3098.233352] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3115.072366] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3124.295549] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3158.795378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.010940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3159.045274] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3159.100969] Lustre: lustre-MDT0000: new disk, initializing [ 3159.154782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.170779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3163.383116] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3173.553377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.660329] Lustre: 111989:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3173.710808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3173.715548] Lustre: Skipped 1 previous similar message [ 3173.771529] Lustre: lustre-MDT0001: new disk, initializing [ 3173.858432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3173.897336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3173.912178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3178.620935] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3186.846169] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3190.808937] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3197.482738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.976110] Lustre: lustre-OST0000: new disk, initializing [ 3197.988228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3198.002383] Lustre: 113730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3198.161243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3203.643035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3203.660283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3203.782892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3205.308720] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3212.750318] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3217.983722] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3218.178576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3222.540629] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3222.764564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3229.668989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3229.675165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.692853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3229.703392] Lustre: Skipped 2 previous similar messages [ 3234.787989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.801391] Lustre: Skipped 1 previous similar message [ 3238.879556] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.176922] Lustre: server umount lustre-OST0000 complete [ 3245.025662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.044080] Lustre: Skipped 2 previous similar messages [ 3245.045060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.054520] Lustre: Skipped 1 previous similar message [ 3257.313224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3257.500162] Lustre: server umount lustre-MDT0000 complete [ 3259.360654] LustreError: 112000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.389843] LustreError: 112000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3261.169915] LustreError: 111980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872856 with bad export cookie 9021148374627938199 [ 3261.174616] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.179695] LustreError: 111980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3261.617187] Lustre: server umount lustre-MDT0001 complete [ 3271.475903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.120605] LustreError: 115804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.235853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.281714] LustreError: 115805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.033420] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3282.400590] LustreError: 115804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.546586] LustreError: 115805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.573222] LustreError: 115805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3289.394897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.972644] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3299.805309] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.417781] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3311.844272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3319.113838] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3324.493327] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3331.505348] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3337.909901] Lustre: server umount lustre-OST0000 complete [ 3343.843560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.847298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.864182] Lustre: Skipped 1 previous similar message [ 3343.902400] Lustre: Skipped 5 previous similar messages [ 3354.085598] LustreError: 115799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.119528] LustreError: 115799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3354.195955] Lustre: server umount lustre-MDT0000 complete [ 3355.118104] Lustre: 101185:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779872933/real 1779872933] req@000000006682fb00 x1866331509081472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1779872949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3356.132230] Lustre: 101183:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779872934/real 1779872934] req@0000000054107460 x1866331509081856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779872950 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3358.386184] LustreError: 118052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872953 with bad export cookie 9021148374627939081 [ 3358.386879] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3358.399211] LustreError: 118052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3358.987784] Lustre: server umount lustre-MDT0001 complete [ 3369.056909] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3371.916125] Key type lgssc unregistered [ 3372.248763] LNet: 119077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.256513] LNetError: 119077:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3372.272449] LNet: Removed LNI 192.168.201.141@tcp [ 3373.002729] Key type .llcrypt unregistered [ 3373.004413] Key type ._llcrypt unregistered [ 3391.571712] Key type ._llcrypt registered [ 3391.576859] Key type .llcrypt registered [ 3391.684413] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3408.448745] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3410.085452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3410.104311] alg: No test for adler32 (adler32-zlib) [ 3411.226551] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 3411.493841] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3413.200903] Key type lgssc registered [ 3414.337615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3448.568307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3448.609487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.019680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3450.055231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3450.158087] Lustre: lustre-MDT0000: new disk, initializing [ 3450.275228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.349216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3455.046048] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3465.939666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.116342] Lustre: 122197:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3466.195728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3466.202672] Lustre: Skipped 1 previous similar message [ 3466.359386] Lustre: lustre-MDT0001: new disk, initializing [ 3466.519503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3466.603877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3466.616160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3471.902325] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3479.693854] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3484.320606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3491.248164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.727849] Lustre: lustre-OST0000: new disk, initializing [ 3491.734803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3491.740937] Lustre: 123939:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3491.858291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.541651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3497.548504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3497.665022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3498.535756] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3507.066385] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3513.120743] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3513.410591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3519.087853] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3519.380778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3523.554993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.563792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3523.564304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3523.576724] Lustre: Skipped 1 previous similar message [ 3527.702981] Lustre: server umount lustre-OST0000 complete [ 3533.281755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3533.289818] LustreError: Skipped 1 previous similar message [ 3533.297358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3533.324752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3533.331058] Lustre: Skipped 1 previous similar message [ 3538.455449] Lustre: server umount lustre-MDT0000 complete [ 3542.739040] LustreError: 122188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873137 with bad export cookie 2258100958606290306 [ 3542.743376] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3542.752858] LustreError: 122188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3543.049511] Lustre: server umount lustre-MDT0001 complete [ 3552.684655] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:12:25 (1779873145) [ 3559.502845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.873538] LustreError: 126206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.892259] LustreError: 126206:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3559.964386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.326140] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3565.026623] LustreError: 126207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.149788] LustreError: 126206:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3572.635304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.008381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3577.455868] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3583.362557] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3587.747503] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3590.626354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3590.631033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3590.649034] Lustre: Skipped 2 previous similar messages [ 3590.662476] Lustre: Skipped 2 previous similar messages [ 3595.745978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3595.762644] Lustre: Skipped 1 previous similar message [ 3596.235364] Lustre: server umount lustre-MDT0000 complete [ 3597.798778] LustreError: 126202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.820525] LustreError: 126202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3599.953535] LustreError: 126187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873194 with bad export cookie 2258100958606291188 [ 3599.958547] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3599.981586] LustreError: 126187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3600.327297] Lustre: server umount lustre-MDT0001 complete [ 3609.997576] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:13:23 (1779873203) [ 3615.941462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3631.519720] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3636.639441] LustreError: 128394:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 3662.367593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3669.714979] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3675.800821] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3678.335434] Lustre: server umount lustre-OST0000 complete [ 3688.906578] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:14:41 (1779873281) [ 3697.752814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.351333] LustreError: 129728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3698.443377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.879903] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3708.899604] LustreError: 129729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.907373] LustreError: 129729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3715.590200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.614633] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3728.815799] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3734.051776] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3741.370841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3741.723143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.728421] Lustre: Skipped 1 previous similar message [ 3749.200828] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3755.933720] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3771.363265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.367854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.393051] Lustre: Skipped 2 previous similar messages [ 3771.397188] Lustre: Skipped 1 previous similar message [ 3776.220171] Lustre: server umount lustre-MDT0000 complete [ 3776.487114] LustreError: 129725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3776.506667] LustreError: 129725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3779.676585] LustreError: 129708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873374 with bad export cookie 2258100958606291853 [ 3779.679231] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3779.684135] LustreError: 129708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3780.123396] Lustre: server umount lustre-MDT0001 complete [ 3794.797974] Lustre: server umount lustre-OST0000 complete [ 3802.888541] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:16:36 (1779873396) [ 3810.225861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.970593] LustreError: 133063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3811.131902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.582092] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3824.129805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.809613] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3835.506884] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3839.123901] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3844.834204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.217342] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3855.545388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3858.217287] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3863.165393] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3863.333976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3868.064682] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3868.370416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3871.202433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.212596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.214352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.214360] Lustre: Skipped 1 previous similar message [ 3871.218511] Lustre: Skipped 1 previous similar message [ 3876.675680] Lustre: server umount lustre-OST0000 complete [ 3880.946145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.957527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.560193] LustreError: 133058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.580247] LustreError: 133058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3886.825923] Lustre: server umount lustre-MDT0000 complete [ 3892.082361] LustreError: 133044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873486 with bad export cookie 2258100958606292924 [ 3892.089445] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.101652] LustreError: 133044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3892.553949] Lustre: server umount lustre-MDT0001 complete [ 3903.743938] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:18:16 (1779873496) [ 3912.582772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3928.287747] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3933.407606] LustreError: 136850:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 3959.137951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.150098] Lustre: Skipped 2 previous similar messages [ 3967.142985] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3974.589956] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3981.536480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.316608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3986.744572] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3996.381895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.288691] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4010.084691] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4016.049974] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4022.777213] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4023.083069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4028.320924] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4028.510195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4032.482393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4032.491615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4032.502515] Lustre: Skipped 1 previous similar message [ 4032.508692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4032.514187] Lustre: Skipped 6 previous similar messages [ 4036.611591] Lustre: server umount lustre-MDT0000 complete [ 4038.112311] LustreError: 136876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.120305] LustreError: 136876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4040.596899] LustreError: 136856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873635 with bad export cookie 2258100958606293946 [ 4040.614660] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.616474] LustreError: 136856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4040.945827] Lustre: server umount lustre-MDT0001 complete [ 4054.623740] Lustre: server umount lustre-OST0000 complete [ 4064.463796] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:20:57 (1779873657) [ 4072.142638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4087.842000] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4092.964908] LustreError: 140676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 4124.844196] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4131.305864] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4137.777107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4138.445751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4142.598651] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4149.784977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.774512] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4160.635556] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4164.692911] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4171.201426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4171.442589] Lustre: lustre-OST0001: new disk, initializing [ 4171.446165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4171.450692] Lustre: 143208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4176.930379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4176.944733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4177.021212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4177.253967] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4183.881604] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4187.831934] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4188.051777] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4191.881916] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4192.034715] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4194.786861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4194.792552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.799477] Lustre: Skipped 2 previous similar messages [ 4194.806251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4194.811120] Lustre: Skipped 2 previous similar messages [ 4199.590784] Lustre: server umount lustre-OST0000 complete [ 4217.311563] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4217.436355] Lustre: server umount lustre-OST0001 complete [ 4221.931759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4221.936086] LustreError: Skipped 3 previous similar messages [ 4227.485538] Lustre: server umount lustre-MDT0000 complete [ 4230.805118] LustreError: 143207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873825 with bad export cookie 2258100958606294898 [ 4230.809326] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4230.811343] LustreError: 143207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4231.096638] Lustre: server umount lustre-MDT0001 complete [ 4250.236954] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:24:03 (1779873843) [ 4251.695264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4253.450342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4254.702434] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:24:08 (1779873848) [ 4260.602043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.888706] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4260.922568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4261.092589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.098229] Lustre: Skipped 6 previous similar messages [ 4264.294147] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4271.496208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.573460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4271.597453] Lustre: 146618:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4275.328525] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4279.936878] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4283.397520] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4289.180572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4289.385955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4294.556500] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4297.361311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4300.510785] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4304.777245] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.956978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4308.947559] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4309.101591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4312.554288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.560766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.575187] Lustre: Skipped 5 previous similar messages [ 4312.583126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4312.591013] Lustre: Skipped 11 previous similar messages [ 4325.343498] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4325.492699] Lustre: server umount lustre-OST0000 complete [ 4345.152417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.485933] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4353.590619] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.608262] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4353.621662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.559606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4359.714405] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4359.952981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.141479] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4364.363401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4368.621939] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ad6858b1800.ost_server_uuid 50 [ 4369.943681] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ad6858b1800.ost_server_uuid in FULL state after 0 sec [ 4370.069979] LustreError: 148711:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.41@tcp) returned error from blocking AST (req@00000000d0c812f2 x1866332117395584 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000061dd88e6/0x1f5662dba0b288cc lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.41@tcp remote: 0x8fa232d04430bee0 expref: 5 pid: 146625 timeout: 4470 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4370.120365] LustreError: lustre-MDT0000: A client on nid 192.168.201.41@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4370.140669] LustreError: 146614:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.41@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000061dd88e6/0x1f5662dba0b288cc lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.41@tcp remote: 0x8fa232d04430bee0 expref: 6 pid: 146625 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4379.794086] Lustre: server umount lustre-OST0000 complete [ 4389.321146] Lustre: server umount lustre-MDT0000 complete [ 4389.349823] LustreError: 146629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.371502] LustreError: 146629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4392.249146] LustreError: 146609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779873987 with bad export cookie 2258100958606296081 [ 4392.250911] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.257878] LustreError: 146609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4398.649362] Lustre: server umount lustre-MDT0001 complete [ 4408.760328] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4411.060972] Key type lgssc unregistered [ 4411.237914] LNet: 152092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4411.256032] LNetError: 152092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4412.326544] LNet: Removed LNI 192.168.201.141@tcp [ 4413.060324] Key type .llcrypt unregistered [ 4413.064369] Key type ._llcrypt unregistered [ 4433.020856] Key type ._llcrypt registered [ 4433.024894] Key type .llcrypt registered [ 4433.160514] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4434.382580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4434.398324] alg: No test for adler32 (adler32-zlib) [ 4435.694524] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4435.941666] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4437.719177] Key type lgssc registered [ 4438.898981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4445.975730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4446.000508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4447.428760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.110355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4457.955112] LustreError: 153467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.974171] LustreError: 153467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4460.032866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.387467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4464.916415] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4470.142543] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4473.153608] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4479.897969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.313617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.107522] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4492.623297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4495.438814] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4505.069758] Lustre: Failing over lustre-MDT0000 [ 4507.206702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4507.219063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.233135] LustreError: 153467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4507.306992] Lustre: server umount lustre-MDT0000 complete [ 4508.151751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.187486] Lustre: Skipped 1 previous similar message [ 4509.090283] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.123745] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4513.261385] LustreError: 153461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4513.304598] LustreError: 153461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4518.370750] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.410935] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4520.672881] LustreError: 153446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.41@tcp arrived at 1779874115 with bad export cookie 13904451486448825934 [ 4520.702486] LustreError: 153446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4523.503953] Lustre: 152780:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779874102/real 1779874102] req@0000000047269ed4 x1866333190908032/t0(0) o400->MGC192.168.201.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779874118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4523.543424] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4528.618730] LustreError: 154960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.654365] LustreError: 154960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4530.894196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.730433] LustreError: 152777:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005473f9a4 x1866333190912768/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4534.153919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.217200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4539.363886] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4539.394334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4539.685121] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4542.244973] Lustre: lustre-MDT0000: Denying connection for new client d34f215e-bebf-4556-83c6-6e28056178cd (at 192.168.201.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 4552.160445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.167866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.177121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.723875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.726743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4554.766550] Lustre: Skipped 1 previous similar message [ 4556.615708] Lustre: server umount lustre-OST0000 complete [ 4561.596983] LustreError: 156842:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4561.611518] Lustre: 156207:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4561.621026] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4561.650560] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4561.664274] LustreError: 156207:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008a21b141 x1866333190929280/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4561.684744] LustreError: 156207:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4561.713473] LustreError: 156207:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4561.740232] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 4562.170478] Lustre: server umount lustre-MDT0000 complete [ 4564.967231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.985554] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.000557] LustreError: 153462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4566.633122] LustreError: 155707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874161 with bad export cookie 13904451486448826032 [ 4566.649430] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.654839] LustreError: 155707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4566.655746] LustreError: 152777:0:(import.c:709:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4567.191242] Lustre: server umount lustre-MDT0001 complete [ 4579.978908] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4583.199163] Key type lgssc unregistered [ 4583.526075] LNet: 157577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4583.532835] LNetError: 157577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4583.554144] LNet: Removed LNI 192.168.201.141@tcp [ 4584.647467] Key type .llcrypt unregistered [ 4584.649738] Key type ._llcrypt unregistered [ 4607.911347] Key type ._llcrypt registered [ 4607.913763] Key type .llcrypt registered [ 4608.015921] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4609.334339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4609.352471] alg: No test for adler32 (adler32-zlib) [ 4610.496687] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4610.877869] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4612.575191] Key type lgssc registered [ 4613.704781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4622.073636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4622.100403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4623.629732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.526942] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4634.089426] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4634.115417] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4635.590980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.852371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4640.178567] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4646.004958] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.049672] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4656.739332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4657.024530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4664.335491] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4668.267896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4671.325894] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4678.624655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4678.635887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4678.646716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4678.658949] Lustre: Skipped 1 previous similar message [ 4683.754538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4683.764810] Lustre: Skipped 2 previous similar messages [ 4684.303555] Lustre: server umount lustre-OST0000 complete [ 4692.450231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4692.468559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4692.505243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4693.990905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.396881] Lustre: server umount lustre-MDT0000 complete [ 4699.107224] LustreError: 159683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4699.126534] LustreError: 159683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4699.854635] LustreError: 158954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874294 with bad export cookie 7195283751699736515 [ 4699.869664] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.872465] LustreError: 158954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4700.379334] Lustre: server umount lustre-MDT0001 complete [ 4713.580354] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4717.007228] Key type lgssc unregistered [ 4717.338962] LNet: 162147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4717.345102] LNetError: 162147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4717.362449] LNet: Removed LNI 192.168.201.141@tcp [ 4718.344924] Key type .llcrypt unregistered [ 4718.351243] Key type ._llcrypt unregistered [ 4760.062733] Key type ._llcrypt registered [ 4760.070789] Key type .llcrypt registered [ 4760.241259] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4761.730521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4761.893916] alg: No test for adler32 (adler32-zlib) [ 4763.084061] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4763.469078] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4765.231513] Key type lgssc registered [ 4766.451860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4774.788161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4774.824782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.559473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.471886] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4786.663299] LustreError: 163835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.683248] LustreError: 163835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4789.542831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.932102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4794.647879] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4800.777027] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4804.790713] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4811.443966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4811.827657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.242579] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4822.065735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4824.492774] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.233289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.235704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4832.246412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4832.274412] Lustre: Skipped 1 previous similar message [ 4837.347492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.665412] Lustre: server umount lustre-OST0000 complete [ 4841.448758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4841.458737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4841.476368] Lustre: Skipped 1 previous similar message [ 4841.485135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.488689] Lustre: Skipped 1 previous similar message [ 4842.471122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.585534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4847.600138] Lustre: Skipped 3 previous similar messages [ 4847.839111] Lustre: server umount lustre-MDT0000 complete [ 4851.969875] LustreError: 164553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874446 with bad export cookie 5197495081933977732 [ 4851.983787] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4851.986696] LustreError: 164553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4852.576109] Lustre: server umount lustre-MDT0001 complete [ 4865.156418] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4868.255892] Key type lgssc unregistered [ 4868.526577] LNet: 167013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4868.537114] LNetError: 167013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4869.613632] LNet: Removed LNI 192.168.201.141@tcp [ 4870.624171] Key type .llcrypt unregistered [ 4870.627693] Key type ._llcrypt unregistered [ 4892.442991] Key type ._llcrypt registered [ 4892.445942] Key type .llcrypt registered [ 4892.548983] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4893.835925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4893.866076] alg: No test for adler32 (adler32-zlib) [ 4894.917525] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 4895.291871] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4897.007146] Key type lgssc registered [ 4898.748914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4910.716837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4910.735366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.106253] Lustre: *** cfs_fail_loc=135, val=0*** [ 4912.172473] LustreError: 168522:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 4912.181200] LustreError: 168522:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.141@tcp: cfg command failed: rc = -2 [ 4912.185878] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4912.185878] [ 4912.197144] LustreError: MGC192.168.201.141@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4912.212438] LustreError: 168483:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4912.218284] LustreError: 168483:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 4912.223458] LustreError: 168483:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 4912.229127] LustreError: 168483:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 4912.305114] Lustre: server umount lustre-MDT0000 complete [ 4912.309512] LustreError: 168483:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4918.368511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4918.942363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4923.405466] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4928.484409] LustreError: 168893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.513714] LustreError: 168893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4932.152912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.808614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4937.377178] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4942.812424] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4946.723321] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4953.570735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.574798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4953.588936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.597533] Lustre: Skipped 1 previous similar message [ 4954.952207] Lustre: server umount lustre-MDT0000 complete [ 4957.740402] LustreError: 169604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4957.768408] LustreError: 169604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4958.446894] LustreError: 168873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874553 with bad export cookie 10760016666694179836 [ 4958.473091] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4959.002848] Lustre: server umount lustre-MDT0001 complete [ 4972.224324] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4975.995552] Key type lgssc unregistered [ 4976.548943] LNet: 170969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4976.561827] LNetError: 170969:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4976.590266] LNet: Removed LNI 192.168.201.141@tcp [ 4977.834463] Key type .llcrypt unregistered [ 4977.837969] Key type ._llcrypt unregistered [ 5005.694185] Key type ._llcrypt registered [ 5005.697503] Key type .llcrypt registered [ 5005.885079] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5006.275638] Key type .llcrypt unregistered [ 5006.278235] Key type ._llcrypt unregistered [ 5025.119381] Key type ._llcrypt registered [ 5025.125238] Key type .llcrypt registered [ 5025.284828] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5026.899792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5026.934935] alg: No test for adler32 (adler32-zlib) [ 5028.244910] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5028.772557] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5030.568455] Key type lgssc registered [ 5032.295487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.022571] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.054201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5056.735423] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5061.856069] LustreError: 173069:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 5088.543248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.769348] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5103.052547] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5111.573899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5111.839526] LustreError: 172405:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000033892d7 x1866333812689152/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5112.450253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.562138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5117.937240] LustreError: 173094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.981072] LustreError: 173094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5118.226949] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5123.042568] LustreError: 173095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.073827] LustreError: 173094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.097939] LustreError: 173094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5129.188942] LustreError: 174005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5129.225437] LustreError: 174005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5129.644924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.176778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.909898] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5143.401366] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5148.317758] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5151.536947] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5163.491360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5163.517328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.561493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.051856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.053419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5166.079109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.179390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.190375] Lustre: Skipped 2 previous similar messages [ 5176.295449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5176.306230] Lustre: Skipped 1 previous similar message [ 5177.311144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5177.637142] Lustre: server umount lustre-OST0000 complete [ 5186.538801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.551382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5186.558235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5189.487580] Lustre: server umount lustre-MDT0000 complete [ 5191.648391] LustreError: 174008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5194.177189] LustreError: 173077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874789 with bad export cookie 1433505770923114886 [ 5194.180412] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.198609] LustreError: 173077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5194.734989] Lustre: server umount lustre-MDT0001 complete [ 5206.357896] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5210.460576] Key type lgssc unregistered [ 5210.982721] LNet: 176689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5210.993036] LNetError: 176689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5211.023226] LNet: Removed LNI 192.168.201.141@tcp [ 5212.213475] Key type .llcrypt unregistered [ 5212.217391] Key type ._llcrypt unregistered [ 5240.002359] Key type ._llcrypt registered [ 5240.004460] Key type .llcrypt registered [ 5240.138867] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5242.505022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5242.535388] alg: No test for adler32 (adler32-zlib) [ 5243.688915] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5244.120093] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5245.839189] Key type lgssc registered [ 5247.030676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5254.020119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5254.050913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.594649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.406991] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5265.892054] LustreError: 178087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.910076] LustreError: 178087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5268.741574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5269.111921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5273.834688] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5280.107614] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5284.230037] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.231898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5291.746534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5298.561114] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5300.981958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5307.065206] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5312.812555] Lustre: Failing over lustre-MDT0000 [ 5313.143977] Lustre: server umount lustre-MDT0000 complete [ 5314.982406] LustreError: 179702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5315.049954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5315.065832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.582687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.584102] LustreError: 178087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5316.597703] Lustre: Skipped 1 previous similar message [ 5316.635621] LustreError: 178087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5320.035297] LustreError: 178084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.163841] LustreError: 178084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.204694] LustreError: 178084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5332.960132] Lustre: 177396:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779874911/real 1779874911] req@00000000c30b4752 x1866334038157696/t0(0) o400->MGC192.168.201.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1779874927 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5332.990891] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.283061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.364592] LustreError: 179702:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5334.381547] LustreError: 179702:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5342.263066] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xf02c18725f241edc [ 5342.270898] Lustre: MGC192.168.201.141@tcp: Connection restored to 0@lo (at 0@lo) [ 5342.551642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.598899] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5345.630816] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5346.584623] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5347.842917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5347.873374] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5347.952244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5351.037662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5353.171349] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5368.294919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.300886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.324456] Lustre: Skipped 1 previous similar message [ 5368.344175] Lustre: Skipped 1 previous similar message [ 5372.594800] Lustre: server umount lustre-OST0000 complete [ 5378.535136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.538711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.565057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.857146] Lustre: server umount lustre-MDT0000 complete [ 5383.648102] LustreError: 178088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.675823] LustreError: 178088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5387.038951] LustreError: 178069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779874981 with bad export cookie 17306234347579514588 [ 5387.046127] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5387.054927] LustreError: 178069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5387.596246] Lustre: server umount lustre-MDT0001 complete [ 5399.316170] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5402.109814] Key type lgssc unregistered [ 5402.490773] LNet: 182474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5402.501457] LNetError: 182474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5402.523331] LNet: Removed LNI 192.168.201.141@tcp [ 5403.500917] Key type .llcrypt unregistered [ 5403.506956] Key type ._llcrypt unregistered [ 5429.223731] Key type ._llcrypt registered [ 5429.225617] Key type .llcrypt registered [ 5429.319221] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5430.495395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5430.690143] alg: No test for adler32 (adler32-zlib) [ 5431.834972] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5432.052768] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5433.719387] Key type lgssc registered [ 5434.897326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5442.235654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5442.259758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.821501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5447.660747] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5453.285746] LustreError: 183876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.313674] LustreError: 183876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5456.195839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.550870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5460.165176] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5466.442592] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5471.015367] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5477.752100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5478.062816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5484.008119] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5489.396393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5489.902676] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5494.075963] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5498.669865] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5510.971684] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5525.473142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.494831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.545712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5526.555102] Lustre: server umount lustre-OST0000 complete [ 5533.696896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.720115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5538.272979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5538.295951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.358661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5539.995493] Lustre: server umount lustre-MDT0000 complete [ 5543.904815] LustreError: 183876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.937555] LustreError: 183876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5544.934126] Lustre: 183188:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779875123/real 1779875123] req@000000004e3d3630 x1866334235313152/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1779875139 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5544.997695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.033209] LustreError: 183875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.084945] LustreError: 183875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5546.628899] LustreError: 186204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875141 with bad export cookie 14207831141081809088 [ 5546.661432] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5547.155589] Lustre: server umount lustre-MDT0001 complete [ 5560.528446] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5564.401967] Key type lgssc unregistered [ 5564.855133] LNet: 187259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5564.864632] LNetError: 187259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5564.902072] LNet: Removed LNI 192.168.201.141@tcp [ 5566.237918] Key type .llcrypt unregistered [ 5566.242590] Key type ._llcrypt unregistered [ 5594.859535] Key type ._llcrypt registered [ 5594.864116] Key type .llcrypt registered [ 5594.953237] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5596.385276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5596.456743] alg: No test for adler32 (adler32-zlib) [ 5597.730672] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5598.142301] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5599.919218] Key type lgssc registered [ 5601.179687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5610.583081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5610.625032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5612.385751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.083292] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5622.763749] LustreError: 188655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.790855] LustreError: 188655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5627.469607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.867532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5633.131125] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5639.023754] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5643.144476] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5650.576580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.122235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.277440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5658.819906] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5666.597456] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5672.861191] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5684.485779] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5699.918766] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5714.398521] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5724.130199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.134342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.146514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.178076] Lustre: Skipped 1 previous similar message [ 5729.252184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5729.267980] Lustre: Skipped 2 previous similar messages [ 5729.461425] Lustre: server umount lustre-OST0000 complete [ 5734.371758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.382834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.391942] Lustre: Skipped 1 previous similar message [ 5734.405932] Lustre: Skipped 1 previous similar message [ 5739.493134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5739.500325] Lustre: Skipped 1 previous similar message [ 5740.235766] Lustre: server umount lustre-MDT0000 complete [ 5744.608976] LustreError: 190272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.630128] LustreError: 190272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5744.766365] LustreError: 191391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875339 with bad export cookie 16897912533905135248 [ 5744.776468] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5744.782051] LustreError: 191391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5745.368362] Lustre: server umount lustre-MDT0001 complete [ 5759.662981] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5764.139319] Key type lgssc unregistered [ 5764.416054] LNet: 193469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5764.429956] LNetError: 193469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5764.478171] LNet: Removed LNI 192.168.201.141@tcp [ 5765.382978] Key type .llcrypt unregistered [ 5765.384888] Key type ._llcrypt unregistered [ 5794.279649] Key type ._llcrypt registered [ 5794.282081] Key type .llcrypt registered [ 5794.367049] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5796.239097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5796.285056] alg: No test for adler32 (adler32-zlib) [ 5797.396947] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5797.663391] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5799.351825] Key type lgssc registered [ 5800.933782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5809.753929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5809.766564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.284715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.374364] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5821.423495] LustreError: 194867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.436868] LustreError: 194867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5826.529166] LustreError: 194866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.579833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.952375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5831.364184] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5837.445817] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5841.872322] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5849.929193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5850.457872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.638851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5859.071044] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5866.523629] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5887.465579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5887.466460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.487666] LustreError: Skipped 1 previous similar message [ 5887.489222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5887.514518] Lustre: Skipped 1 previous similar message [ 5887.558905] Lustre: Skipped 1 previous similar message [ 5890.285178] Lustre: server umount lustre-OST0000 complete [ 5897.696387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.702358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5897.727867] Lustre: Skipped 1 previous similar message [ 5901.113275] Lustre: server umount lustre-MDT0000 complete [ 5902.821315] LustreError: 194867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5902.838714] LustreError: 194867:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5905.615484] LustreError: 195584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875500 with bad export cookie 7025746382385047751 [ 5905.624710] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5905.643460] LustreError: 195584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5906.564593] Lustre: server umount lustre-MDT0001 complete [ 5919.297144] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5922.605393] Key type lgssc unregistered [ 5922.899115] LNet: 198196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5922.911375] LNetError: 198196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5922.928650] LNet: Removed LNI 192.168.201.141@tcp [ 5924.022356] Key type .llcrypt unregistered [ 5924.025247] Key type ._llcrypt unregistered [ 5947.701841] Key type ._llcrypt registered [ 5947.703445] Key type .llcrypt registered [ 5947.894425] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5949.034761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5949.236266] alg: No test for adler32 (adler32-zlib) [ 5950.596292] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5950.807124] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5952.487332] Key type lgssc registered [ 5953.803804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5962.492451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5962.511273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.901156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5968.371643] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5973.485780] LustreError: 199448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.508047] LustreError: 199448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5977.376590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.902952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.328069] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5990.901620] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5995.979737] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6003.659195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6004.201960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.304986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6012.092522] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6018.475367] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6037.987511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.989980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6038.004393] Lustre: Skipped 1 previous similar message [ 6038.017473] Lustre: Skipped 1 previous similar message [ 6041.800764] Lustre: server umount lustre-OST0000 complete [ 6048.224746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6048.227967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6048.237299] Lustre: Skipped 1 previous similar message [ 6053.129356] Lustre: server umount lustre-MDT0000 complete [ 6053.366163] LustreError: 199449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.390893] LustreError: 199449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6056.816195] LustreError: 199429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875651 with bad export cookie 11165725662378011309 [ 6056.821812] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6056.837270] LustreError: 199429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6057.110317] Lustre: server umount lustre-MDT0001 complete [ 6069.089456] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6072.496317] Key type lgssc unregistered [ 6072.891666] LNet: 202775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6072.901233] LNetError: 202775:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6072.940663] LNet: Removed LNI 192.168.201.141@tcp [ 6073.894766] Key type .llcrypt unregistered [ 6073.897823] Key type ._llcrypt unregistered [ 6095.431527] Key type ._llcrypt registered [ 6095.433943] Key type .llcrypt registered [ 6095.547665] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6096.660356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6096.671355] alg: No test for adler32 (adler32-zlib) [ 6097.960920] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6098.294829] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6100.055284] Key type lgssc registered [ 6101.705628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6110.992137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6111.011925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.888276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6118.305977] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6122.475964] LustreError: 204041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.496760] LustreError: 204041:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6127.591306] LustreError: 204040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.154738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6128.941397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6133.859323] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6140.999962] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6145.077129] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6151.983070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6152.317454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6157.448381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6158.914914] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6164.582635] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6175.035632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.127605] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6175.426652] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6182.899796] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6188.786807] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6190.206499] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6202.139724] Lustre: Permanently deactivating lustre-OST0001 [ 6202.155445] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6211.670029] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6222.479590] Lustre: Permanently reactivating lustre-OST0001 [ 6222.501855] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6222.508355] Lustre: Skipped 2 previous similar messages [ 6229.658781] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.694417] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6229.708668] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6229.746628] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6229.865964] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6235.105577] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.111132] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.128385] LustreError: Skipped 1 previous similar message [ 6235.150117] Lustre: Skipped 1 previous similar message [ 6235.170648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6237.161353] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6237.200451] Lustre: Skipped 2 previous similar messages [ 6240.439386] Lustre: server umount lustre-OST0001 complete [ 6242.298191] LustreError: 205637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6242.318037] LustreError: 205637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6244.833044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6244.847561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.897637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.864879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.396379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.398933] Lustre: Skipped 2 previous similar messages [ 6247.407480] LustreError: 205637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.464539] LustreError: 205637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6250.974797] Lustre: server umount lustre-OST0000 complete [ 6257.120440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6257.134257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.156925] Lustre: Skipped 1 previous similar message [ 6257.165567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.844809] Lustre: server umount lustre-MDT0000 complete [ 6262.760213] LustreError: 204036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.795118] LustreError: 204036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6266.056288] LustreError: 207212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875860 with bad export cookie 1148965868976051270 [ 6266.058045] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6266.090340] LustreError: 207212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6266.606466] Lustre: server umount lustre-MDT0001 complete [ 6278.621974] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6281.882574] Key type lgssc unregistered [ 6282.261764] LNet: 208669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6282.269377] LNetError: 208669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6283.302410] LNet: Removed LNI 192.168.201.141@tcp [ 6284.341479] Key type .llcrypt unregistered [ 6284.343806] Key type ._llcrypt unregistered [ 6327.622534] Key type ._llcrypt registered [ 6327.624627] Key type .llcrypt registered [ 6327.715970] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6328.959437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6329.014067] alg: No test for adler32 (adler32-zlib) [ 6330.094574] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6330.268745] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6331.927255] Key type lgssc registered [ 6333.508874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6340.598644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6340.622984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6341.928699] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6341.961258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6342.159117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6346.467829] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6357.290098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6357.422405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6357.458189] Lustre: 210862:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6357.644695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.768394] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6367.404303] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6371.183983] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6378.203517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.429476] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6378.622825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.353351] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6386.415470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6389.983390] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6392.670173] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6400.515333] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6408.040721] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6415.985405] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6430.803616] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6430.806160] Lustre: Skipped 1 previous similar message [ 6449.018097] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6449.024717] Lustre: Skipped 1 previous similar message [ 6486.640022] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6486.646506] Lustre: Skipped 4 previous similar messages [ 6556.311038] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6556.317913] Lustre: Skipped 8 previous similar messages [ 6565.859538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6565.863638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.880915] LustreError: Skipped 1 previous similar message [ 6565.891490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6565.915184] Lustre: Skipped 1 previous similar message [ 6569.261423] Lustre: server umount lustre-OST0000 complete [ 6576.100385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6576.117094] Lustre: Skipped 1 previous similar message [ 6576.138267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.148818] Lustre: Skipped 1 previous similar message [ 6581.220160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6581.242402] Lustre: Skipped 1 previous similar message [ 6585.323077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6585.333781] Lustre: Skipped 1 previous similar message [ 6588.383240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6588.776079] Lustre: server umount lustre-MDT0000 complete [ 6593.121091] LustreError: 210853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876187 with bad export cookie 7788296851303881313 [ 6593.123920] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.129046] LustreError: 210853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6593.418869] Lustre: server umount lustre-MDT0001 complete [ 6609.148797] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6613.484734] Key type lgssc unregistered [ 6613.829904] LNet: 215321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6613.840224] LNetError: 215321:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6613.856065] LNet: Removed LNI 192.168.201.141@tcp [ 6615.108655] Key type .llcrypt unregistered [ 6615.112226] Key type ._llcrypt unregistered [ 6645.905242] Key type ._llcrypt registered [ 6645.917468] Key type .llcrypt registered [ 6646.224481] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6648.079016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6648.130288] alg: No test for adler32 (adler32-zlib) [ 6649.104338] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6649.235754] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6651.023258] Key type lgssc registered [ 6653.116972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6662.664248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6662.686905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6664.320874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.519412] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6674.914488] LustreError: 216722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.933719] LustreError: 216722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6680.037971] LustreError: 216723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6680.187965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.690801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6686.299196] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6692.396763] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6696.175724] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6703.039354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.497053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.631669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6711.182331] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6719.379048] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6740.448418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6740.458678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6740.473917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.478453] Lustre: Skipped 1 previous similar message [ 6740.487788] Lustre: Skipped 1 previous similar message [ 6745.581085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6745.585372] Lustre: Skipped 1 previous similar message [ 6750.693403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6750.700760] Lustre: Skipped 1 previous similar message [ 6754.272443] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6754.588054] Lustre: server umount lustre-OST0000 complete [ 6760.931423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.956854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6760.973599] Lustre: Skipped 1 previous similar message [ 6765.708044] Lustre: server umount lustre-MDT0000 complete [ 6766.048219] LustreError: 216722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6766.074868] LustreError: 216722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6770.391481] LustreError: 216702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876365 with bad export cookie 6466007081470238940 [ 6770.398400] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.407048] LustreError: 216702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6770.947286] Lustre: server umount lustre-MDT0001 complete [ 6784.243265] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6787.648783] Key type lgssc unregistered [ 6787.890608] LNet: 220058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6787.897489] LNetError: 220058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6787.912561] LNet: Removed LNI 192.168.201.141@tcp [ 6789.420449] Key type .llcrypt unregistered [ 6789.422259] Key type ._llcrypt unregistered [ 6816.012969] Key type ._llcrypt registered [ 6816.015129] Key type .llcrypt registered [ 6816.111995] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6817.306748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6817.365820] alg: No test for adler32 (adler32-zlib) [ 6818.525984] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6818.775806] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6820.423212] Key type lgssc registered [ 6821.615644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7095.676090] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7098.930954] Key type lgssc unregistered [ 7099.234578] LNet: 222048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7099.253618] LNetError: 222048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7099.275490] LNet: Removed LNI 192.168.201.141@tcp [ 7100.309823] Key type .llcrypt unregistered [ 7100.313045] Key type ._llcrypt unregistered [ 7127.240401] Key type ._llcrypt registered [ 7127.243412] Key type .llcrypt registered [ 7127.362811] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7144.964399] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7146.641733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7146.725962] alg: No test for adler32 (adler32-zlib) [ 7148.020141] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 7148.341062] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7150.135193] Key type lgssc registered [ 7151.425193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7186.302964] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6881 sec ========== 06:12:59 (1779876779) [ 7188.675955] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:13:01 (1779876781) === [ 7193.265651] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:13:05 (1779876785) === [ 7241.301656] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7244.679276] Key type lgssc unregistered [ 7244.935233] LNet: 227233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7244.942323] LNetError: 227233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7244.974676] LNet: Removed LNI 192.168.201.141@tcp [ 7246.047681] Key type .llcrypt unregistered [ 7246.049533] Key type ._llcrypt unregistered