[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494345497 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002293] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006874] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009104] LSM: Security Framework initializing [ 0.010043] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.019352] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024338] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027132] *** VALIDATE tmpfs *** [ 0.028467] *** VALIDATE proc *** [ 0.030257] *** VALIDATE cgroup *** [ 0.031011] *** VALIDATE cgroup2 *** [ 0.033075] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036038] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040079] debug: unmapping init [mem 0xffffffff9b859000-0xffffffff9b860fff] [ 0.042938] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044033] ... version: 2 [ 0.045019] ... bit width: 48 [ 0.046027] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048023] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050016] ... event mask: 000000070000000f [ 0.051336] rcu: Hierarchical SRCU implementation. [ 0.053438] smp: Bringing up secondary CPUs ... [ 0.054517] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.058687] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142475] node 0 deferred pages initialised in 79ms [ 0.146009] devtmpfs: initialized [ 0.147253] x86/mm: Memory block size: 128MB [ 0.149840] gcov: version magic: 0x41383552 [ 0.151011] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155490] pinctrl core: initialized pinctrl subsystem [ 0.156140] [ 0.156469] ************************************************************* [ 0.159010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160011] ** ** [ 0.163016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165014] ** ** [ 0.168017] ** This means that this kernel is built to expose internal ** [ 0.170228] ** IOMMU data structures, which may compromise security on ** [ 0.173017] ** your system. ** [ 0.175014] ** ** [ 0.176010] ** If you see this message and you are not debugging the ** [ 0.178009] ** kernel, report this immediately to your vendor! ** [ 0.179009] ** ** [ 0.181010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182007] ************************************************************* [ 0.184667] NET: Registered protocol family 16 [ 0.186405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.190047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192032] cpuidle: using governor menu [ 0.193733] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195366] PCI: Using configuration type 1 for base access [ 0.196096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206144] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208103] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211155] cryptd: max_cpu_qlen set to 1000 [ 0.214302] ACPI: Added _OSI(Module Device) [ 0.216014] ACPI: Added _OSI(Processor Device) [ 0.217010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223067] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.228008] ACPI: Interpreter enabled [ 0.230050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.231025] ACPI: Using IOAPIC for interrupt routing [ 0.232173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.235554] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245327] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.248056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.253098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259144] acpiphp: Slot [2] registered [ 0.261129] acpiphp: Slot [5] registered [ 0.262083] acpiphp: Slot [6] registered [ 0.263095] acpiphp: Slot [7] registered [ 0.264098] acpiphp: Slot [8] registered [ 0.265085] acpiphp: Slot [9] registered [ 0.266108] acpiphp: Slot [10] registered [ 0.267089] acpiphp: Slot [3] registered [ 0.268071] acpiphp: Slot [4] registered [ 0.269120] acpiphp: Slot [11] registered [ 0.270083] acpiphp: Slot [12] registered [ 0.271096] acpiphp: Slot [13] registered [ 0.272104] acpiphp: Slot [14] registered [ 0.273075] acpiphp: Slot [15] registered [ 0.273992] acpiphp: Slot [16] registered [ 0.275057] acpiphp: Slot [17] registered [ 0.276050] acpiphp: Slot [18] registered [ 0.277108] acpiphp: Slot [19] registered [ 0.278145] acpiphp: Slot [20] registered [ 0.279131] acpiphp: Slot [21] registered [ 0.281106] acpiphp: Slot [22] registered [ 0.282091] acpiphp: Slot [23] registered [ 0.284101] acpiphp: Slot [24] registered [ 0.285102] acpiphp: Slot [25] registered [ 0.286057] acpiphp: Slot [26] registered [ 0.287031] acpiphp: Slot [27] registered [ 0.288044] acpiphp: Slot [28] registered [ 0.288964] acpiphp: Slot [29] registered [ 0.290088] acpiphp: Slot [30] registered [ 0.290895] acpiphp: Slot [31] registered [ 0.291039] PCI host bridge to bus 0000:00 [ 0.292015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.294023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.300019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301142] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.302779] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.304899] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.311998] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.315039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.319012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.320013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.322437] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323567] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328194] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.331016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.342017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345971] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.351957] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.364020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.378026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.393016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.405018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.412000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.419016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.426019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.446019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.457258] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.464013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.477027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.491018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.501000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.509016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.515017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.527016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.535161] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.542023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.549021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.562014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.571611] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.573387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.580404] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.583394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.587284] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.593118] iommu: Default domain type: Passthrough [ 0.594425] SCSI subsystem initialized [ 0.595141] ACPI: bus type USB registered [ 0.597121] usbcore: registered new interface driver usbfs [ 0.598081] usbcore: registered new interface driver hub [ 0.599073] usbcore: registered new device driver usb [ 0.601166] pps_core: LinuxPPS API ver. 1 registered [ 0.602010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.604047] PTP clock support registered [ 0.606083] EDAC MC: Ver: 3.0.0 [ 0.607679] PCI: Using ACPI for IRQ routing [ 0.609049] NetLabel: Initializing [ 0.610009] NetLabel: domain hash size = 128 [ 0.612011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.613080] NetLabel: unlabeled traffic allowed by default [ 0.615128] vgaarb: loaded [ 0.616172] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.618010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.622456] clocksource: Switched to clocksource kvm-clock [ 0.723587] VFS: Disk quotas dquot_6.6.0 [ 0.725300] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.728124] *** VALIDATE ramfs *** [ 0.729510] *** VALIDATE hugetlbfs *** [ 0.730671] pnp: PnP ACPI init [ 0.732443] pnp: PnP ACPI: found 6 devices [ 0.746993] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.750496] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.752899] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.755316] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.758041] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.760619] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.763713] NET: Registered protocol family 2 [ 0.766490] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.772199] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.775179] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.780090] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.782849] TCP: Hash tables configured (established 65536 bind 65536) [ 0.785910] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.789047] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.792077] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794601] NET: Registered protocol family 1 [ 0.796901] RPC: Registered named UNIX socket transport module. [ 0.798312] RPC: Registered udp transport module. [ 0.799291] RPC: Registered tcp transport module. [ 0.800559] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.802455] NET: Registered protocol family 44 [ 0.804096] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.805810] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.808029] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.810280] PCI: CLS 0 bytes, default 64 [ 0.811896] Unpacking initramfs... [ 2.172570] debug: unmapping init [mem 0xffff99c9fcc54000-0xffff99c9fffbffff] [ 2.175721] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.177419] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.179160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.621038] Initialise system trusted keyrings [ 2.622793] Key type blacklist registered [ 2.624750] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.631382] zbud: loaded [ 2.634085] *** VALIDATE nfs *** [ 2.635371] *** VALIDATE nfs4 *** [ 2.637110] pstore: using deflate compression [ 2.640064] Platform Keyring initialized [ 2.736993] NET: Registered protocol family 38 [ 2.738874] Key type asymmetric registered [ 2.740471] Asymmetric key parser 'x509' registered [ 2.742271] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.745492] io scheduler mq-deadline registered [ 2.747364] io scheduler kyber registered [ 2.749031] io scheduler bfq registered [ 2.750817] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.753516] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.756349] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.759318] ACPI: Power Button [PWRF] [ 2.764557] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.771509] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.781977] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.788224] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.801471] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.827653] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.855811] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.860852] Non-volatile memory driver v1.3 [ 2.862470] Linux agpgart interface v0.103 [ 2.894917] virtio_blk virtio1: [vda] 146856 512-byte logical blocks (75.2 MB/71.7 MiB) [ 2.897571] vda: detected capacity change from 0 to 75190272 [ 2.912115] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.914753] vdb: detected capacity change from 0 to 1073741824 [ 2.928941] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.931653] vdc: detected capacity change from 0 to 2621440000 [ 2.943932] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.945769] vdd: detected capacity change from 0 to 2621440000 [ 2.959571] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.962717] vde: detected capacity change from 0 to 4294967296 [ 2.981690] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.984590] vdf: detected capacity change from 0 to 4294967296 [ 3.000131] libphy: Fixed MDIO Bus: probed [ 3.012685] usbcore: registered new interface driver usbserial_generic [ 3.014513] usbserial: USB Serial support registered for generic [ 3.016206] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.020667] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.022191] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.024820] mousedev: PS/2 mouse device common for all mice [ 3.027489] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.028434] rtc_cmos 00:05: RTC can wake from S4 [ 3.033177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.036734] rtc_cmos 00:05: registered as rtc0 [ 3.039885] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.042152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.043055] intel_pstate: CPU model not supported [ 3.048851] hid: raw HID events driver (C) Jiri Kosina [ 3.051219] usbcore: registered new interface driver usbhid [ 3.053347] usbhid: USB HID core driver [ 3.054964] drop_monitor: Initializing network drop monitor service [ 3.057364] Initializing XFRM netlink socket [ 3.059540] NET: Registered protocol family 10 [ 3.063115] Segment Routing with IPv6 [ 3.063967] NET: Registered protocol family 17 [ 3.065882] mpls_gso: MPLS GSO support [ 3.070930] RAS: Correctable Errors collector initialized. [ 3.073200] AVX version of gcm_enc/dec engaged. [ 3.075081] AES CTR mode by8 optimization enabled [ 3.140409] sched_clock: Marking stable (3140381172, 0)->(4096544204, -956163032) [ 3.143820] registered taskstats version 1 [ 3.145751] Loading compiled-in X.509 certificates [ 3.147270] zswap: loaded using pool lzo/zbud [ 3.169686] Key type big_key registered [ 3.182679] Key type encrypted registered [ 3.184049] ima: No TPM chip found, activating TPM-bypass! [ 3.186089] ima: Allocated hash algorithm: sha1 [ 3.188058] ima: No architecture policies found [ 3.189902] evm: Initialising EVM extended attributes: [ 3.191831] evm: security.selinux [ 3.193052] evm: security.ima [ 3.194104] evm: security.capability [ 3.195412] evm: HMAC attrs: 0x1 [ 3.197600] rtc_cmos 00:05: setting system clock to 2026-08-27 09:38:52 UTC (1787823532) [ 3.203770] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.206379] debug: unmapping init [mem 0xffffffff9b582000-0xffffffff9b858fff] [ 3.215102] Write protecting the kernel read-only data: 28672k [ 3.218712] debug: unmapping init [mem 0xffffffff99c03000-0xffffffff99dfffff] [ 3.221994] debug: unmapping init [mem 0xffffffff9a514000-0xffffffff9a5fffff] [ 3.253526] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.259463] systemd[1]: Detected virtualization kvm. [ 3.260534] systemd[1]: Detected architecture x86-64. [ 3.261717] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.283308] systemd[1]: No hostname configured. [ 3.285011] systemd[1]: Set hostname to . [ 3.286853] random: systemd: uninitialized urandom read (16 bytes read) [ 3.289126] systemd[1]: Initializing machine ID from random generator. [ 3.403374] random: systemd: uninitialized urandom read (16 bytes read) [ 3.406066] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.409783] random: systemd: uninitialized urandom read (16 bytes read) [ 3.413089] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.417380] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.921328] device-mapper: uevent: version 1.0.3 [ 3.923658] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.583236] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook...[ 4.594307] random: fast init done [ 4.653990] scsi host0: ata_piix [ 4.667534] scsi host1: ata_piix [ 4.673698] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.675934] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.730218] random: crng init done [ 9.739385] random: 7 urandom warning(s) missed due to ratelimiting [ 10.162855] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.666380] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.833624] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.890659] hrtimer: interrupt took 8014106 ns [ 16.754433] SELinux: Disabled at runtime. [ 16.875021] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.900269] systemd[1]: Detected virtualization kvm. [ 16.905983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.532636] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.539936] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.551839] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.557748] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.562917] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.568414] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.582633] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 18.805978] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 19.941063] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.534386] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.589087] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.132496] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.233993] EDAC sbridge: Ver: 1.1.2 [ 25.800399] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.551088] NFS: Registering the id_resolver key type [ 26.563832] Key type id_resolver registered [ 26.574389] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 93.917637] libcfs: loading out-of-tree module taints kernel. [ 94.001663] Key type ._llcrypt registered [ 94.007086] Key type .llcrypt registered [ 94.139524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 113.943350] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 115.324437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 115.345867] alg: No test for adler32 (adler32-zlib) [ 116.819390] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 117.464221] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 119.176167] Key type lgssc registered [ 121.996469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.791987] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 189.777874] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 203.299709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 203.352914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.712707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.767703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.873483] Lustre: lustre-MDT0000: new disk, initializing [ 204.962987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.991716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 209.729465] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.306144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.469890] Lustre: 6528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 223.509620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 223.516763] Lustre: Skipped 1 previous similar message [ 223.670618] Lustre: lustre-MDT0001: new disk, initializing [ 223.801619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.846214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.874309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 229.569063] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.089936] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 245.541875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.833713] Lustre: lustre-OST0000: new disk, initializing [ 245.837311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.846338] Lustre: 8464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.955217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 246.837452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 246.859476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.003725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.765791] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.029979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 270.276900] Lustre: lustre-OST0001: new disk, initializing [ 270.291531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 270.299914] Lustre: 9536:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 270.413173] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.050266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 278.068108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 278.162786] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.305625] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.066559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.603515] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 303.629862] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 310.401152] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 315.128591] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 317.618244] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 320.743282] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 322.566809] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 05:44:10 EDT 2026 [ 340.759560] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 342.280657] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.907426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.908973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 359.919546] Lustre: Skipped 1 previous similar message [ 359.931913] Lustre: Skipped 3 previous similar messages [ 365.036676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 365.043164] Lustre: Skipped 3 previous similar messages [ 370.146380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 370.155863] Lustre: Skipped 3 previous similar messages [ 372.192188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 372.368294] Lustre: server umount lustre-MDT0000 complete [ 376.085845] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787823905 with bad export cookie 5465826383059746804 [ 376.093091] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 376.103694] LustreError: 6520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 380.388705] LustreError: 9714:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.389375] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.394928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 380.431439] LustreError: 9714:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 380.464154] Lustre: Skipped 3 previous similar messages [ 381.410044] LustreError: 12181:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.425548] LustreError: 12181:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 382.660880] Lustre: server umount lustre-MDT0001 complete [ 386.602665] Lustre: server umount lustre-OST0000 complete [ 390.437327] Lustre: server umount lustre-OST0001 complete [ 398.135868] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 408.459868] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 448.456382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.726227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 448.749856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 448.857788] Lustre: lustre-MDT0000: new disk, initializing [ 448.937404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.965454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.733931] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 465.737963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.928700] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 466.032242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 466.038980] Lustre: Skipped 1 previous similar message [ 466.160064] Lustre: lustre-MDT0001: new disk, initializing [ 466.269261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 466.332698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 466.365879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 471.563092] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 483.334022] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.924440] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 498.621179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 498.894655] Lustre: lustre-OST0000: new disk, initializing [ 498.898492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 498.903734] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 499.002678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.313492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 500.320111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 500.513702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 506.071907] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 515.669413] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 520.644458] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 520.846291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 525.261994] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 525.453642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 531.428708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.433114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.443526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 531.443535] Lustre: Skipped 3 previous similar messages [ 531.488950] Lustre: Skipped 1 previous similar message [ 533.271497] Lustre: server umount lustre-OST0000 complete [ 536.552046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.665641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 541.683370] Lustre: Skipped 4 previous similar messages [ 543.034290] Lustre: server umount lustre-MDT0000 complete [ 546.786102] LustreError: 15106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.806105] LustreError: 15106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 547.506890] LustreError: 17063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824076 with bad export cookie 5465826383059748337 [ 547.511512] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.520097] LustreError: 17063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 547.978413] Lustre: server umount lustre-MDT0001 complete [ 551.636356] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 05:47:59 (1787824079) [ 558.324837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.777925] LustreError: 19328:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.842696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.949541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 564.195993] LustreError: 19329:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.486930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.893107] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 587.653203] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.686889] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.256540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.124475] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 616.840783] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.200660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.204929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.232308] Lustre: Skipped 1 previous similar message [ 628.242162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 633.678492] Lustre: server umount lustre-OST0000 complete [ 638.438586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.450865] Lustre: Skipped 1 previous similar message [ 643.868125] Lustre: server umount lustre-MDT0000 complete [ 647.820182] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824177 with bad export cookie 5465826383059749219 [ 647.838759] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 647.840753] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 648.218709] Lustre: server umount lustre-MDT0001 complete [ 661.749910] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 664.608667] Key type lgssc unregistered [ 665.050838] LNet: 22865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.062875] LNetError: 22865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.097719] LNet: Removed LNI 192.168.202.132@tcp [ 666.278145] Key type .llcrypt unregistered [ 666.279907] Key type ._llcrypt unregistered [ 690.896964] Key type ._llcrypt registered [ 690.899142] Key type .llcrypt registered [ 691.047867] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 692.338352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.392924] alg: No test for adler32 (adler32-zlib) [ 693.393669] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 693.607936] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 695.321439] Key type lgssc registered [ 696.805835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 705.611497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 705.635192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.239616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 712.822526] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 716.781440] LustreError: 24280:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.800721] LustreError: 24280:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 721.901819] LustreError: 24279:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.846242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.447187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 728.859107] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 736.664259] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 742.848357] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 749.256191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.575881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.709133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 756.219959] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 764.083200] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 781.967391] Lustre: server umount lustre-OST0000 complete [ 788.968236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 788.971665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.008420] Lustre: Skipped 1 previous similar message [ 794.084480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.097064] Lustre: Skipped 2 previous similar messages [ 797.393224] Lustre: server umount lustre-MDT0000 complete [ 799.203335] LustreError: 24280:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.216051] LustreError: 24280:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 800.226368] Lustre: 23592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787824313/real 1787824313] req@0000000013fcb754 x1874669571438720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787824329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 800.227021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.256399] Lustre: 23592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 801.106464] LustreError: 24260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824330 with bad export cookie 2190748940702456109 [ 801.120659] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.558473] Lustre: server umount lustre-MDT0001 complete [ 815.419852] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 819.499905] Key type lgssc unregistered [ 819.800476] LNet: 28258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.809358] LNetError: 28258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.826788] LNet: Removed LNI 192.168.202.132@tcp [ 821.205775] Key type .llcrypt unregistered [ 821.212095] Key type ._llcrypt unregistered [ 844.097168] Key type ._llcrypt registered [ 844.100089] Key type .llcrypt registered [ 844.230586] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 845.710889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.730710] alg: No test for adler32 (adler32-zlib) [ 846.709483] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 846.911907] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 848.505097] Key type lgssc registered [ 849.115455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.603886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.635450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.142135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.928671] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 868.321386] LustreError: 29672:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.355803] LustreError: 29672:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 871.372608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.766116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.635582] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 883.925954] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 889.665283] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 903.611391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 904.012579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 909.608667] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 915.679558] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 928.637415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 938.980245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 939.000599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 939.005336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 939.016406] Lustre: Skipped 1 previous similar message [ 944.104261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 944.108828] Lustre: Skipped 1 previous similar message [ 945.183585] Lustre: server umount lustre-OST0000 complete [ 953.826731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.835572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.853668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.859995] Lustre: Skipped 1 previous similar message [ 955.959138] Lustre: server umount lustre-MDT0000 complete [ 959.463040] LustreError: 29671:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 959.474594] LustreError: 29671:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 959.703844] LustreError: 29652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824489 with bad export cookie 11687491178449135961 [ 959.734463] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.224175] Lustre: server umount lustre-MDT0001 complete [ 971.320885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 974.543838] Key type lgssc unregistered [ 974.976510] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.985870] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.012759] LNet: Removed LNI 192.168.202.132@tcp [ 975.893518] Key type .llcrypt unregistered [ 975.897332] Key type ._llcrypt unregistered [ 1003.475117] Key type ._llcrypt registered [ 1003.477921] Key type .llcrypt registered [ 1003.622569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1005.430379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.455763] alg: No test for adler32 (adler32-zlib) [ 1006.510310] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1006.700189] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1008.345950] Key type lgssc registered [ 1009.677729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1017.164626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1017.188548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.718161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1023.608433] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1028.073217] LustreError: 35002:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.086508] LustreError: 35002:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1031.646679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.967797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1037.025093] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1044.625943] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.860281] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1058.970058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.591772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.665225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1066.629814] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1074.377373] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1088.499422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1088.513196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1088.524630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.609408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.625086] Lustre: Skipped 2 previous similar messages [ 1094.272486] Lustre: server umount lustre-OST0000 complete [ 1098.722170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1098.727662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.733849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.757017] Lustre: Skipped 2 previous similar messages [ 1103.843069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.862545] Lustre: Skipped 2 previous similar messages [ 1104.885203] Lustre: server umount lustre-MDT0000 complete [ 1108.966214] LustreError: 35729:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.979266] LustreError: 35729:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1109.189326] LustreError: 34981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824638 with bad export cookie 10598139887482469749 [ 1109.201292] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.507502] Lustre: server umount lustre-MDT0001 complete [ 1120.033440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1123.011039] Key type lgssc unregistered [ 1123.320041] LNet: 38558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.328202] LNetError: 38558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.348770] LNet: Removed LNI 192.168.202.132@tcp [ 1124.292864] Key type .llcrypt unregistered [ 1124.296383] Key type ._llcrypt unregistered [ 1149.782380] Key type ._llcrypt registered [ 1149.784491] Key type .llcrypt registered [ 1149.912127] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1151.410298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.432089] alg: No test for adler32 (adler32-zlib) [ 1152.519622] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1152.968128] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1154.665897] Key type lgssc registered [ 1155.959863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1163.682281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1163.692336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.064514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1169.768723] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1175.522295] LustreError: 39975:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.541133] LustreError: 39975:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1178.414946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.027427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1183.559400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1191.310522] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1196.776950] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1203.888646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.290565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.435685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1210.545347] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1218.403278] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1225.253394] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1225.700875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1225.703528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.705970] LustreError: Skipped 1 previous similar message [ 1225.730266] Lustre: Skipped 1 previous similar message [ 1230.347063] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1230.351822] Lustre: Skipped 2 previous similar messages [ 1230.797672] Lustre: server umount lustre-OST0000 complete [ 1250.785930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.799340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.824365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.417985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.440606] Lustre: Skipped 3 previous similar messages [ 1257.061923] Lustre: server umount lustre-MDT0000 complete [ 1260.510494] LustreError: 39956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824789 with bad export cookie 11939867852205335676 [ 1260.519503] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.525240] LustreError: 39956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1267.105498] Lustre: server umount lustre-MDT0001 complete [ 1280.517648] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1284.099179] Key type lgssc unregistered [ 1284.405757] LNet: 43685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.415464] LNetError: 43685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.448284] LNet: Removed LNI 192.168.202.132@tcp [ 1285.670469] Key type .llcrypt unregistered [ 1285.675767] Key type ._llcrypt unregistered [ 1311.311536] Key type ._llcrypt registered [ 1311.313434] Key type .llcrypt registered [ 1311.427515] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1312.514160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.624453] alg: No test for adler32 (adler32-zlib) [ 1313.935158] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1314.356859] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1316.080641] Key type lgssc registered [ 1317.419893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.436801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.477644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.227932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1333.613401] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1338.348557] LustreError: 45099:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1338.373259] LustreError: 45099:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1342.278557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.712584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.390916] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1356.467909] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.661632] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1368.967615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.322112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.431764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1378.107770] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1388.263793] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1395.256994] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1397.228760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.258980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.330357] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1400.342600] Lustre: Skipped 2 previous similar messages [ 1401.646406] Lustre: server umount lustre-MDT0000 complete [ 1402.338360] LustreError: 45100:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.367526] LustreError: 45100:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1405.450297] LustreError: 47514:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.936455] LustreError: 45082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787824935 with bad export cookie 18208740068413131181 [ 1405.945064] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1405.951062] LustreError: 45082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1407.457774] LustreError: 45095:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.458826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.482531] Lustre: Skipped 2 previous similar messages [ 1407.486779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1415.650351] LustreError: 45094:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.652077] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1415.663224] LustreError: 45094:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1415.680604] Lustre: Skipped 2 previous similar messages [ 1420.259612] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1420.471386] Lustre: server umount lustre-MDT0001 complete [ 1437.294286] Lustre: server umount lustre-OST0000 complete [ 1457.517229] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1459.973891] Key type lgssc unregistered [ 1460.171719] LNet: 48977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.175897] LNetError: 48977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.187743] LNet: Removed LNI 192.168.202.132@tcp [ 1460.953554] Key type .llcrypt unregistered [ 1460.955310] Key type ._llcrypt unregistered [ 1489.106581] Key type ._llcrypt registered [ 1489.109962] Key type .llcrypt registered [ 1489.220116] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1490.943830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1491.037278] alg: No test for adler32 (adler32-zlib) [ 1492.256740] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1492.756379] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1494.496198] Key type lgssc registered [ 1495.706771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1503.046317] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1503.121681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1518.880730] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1524.000430] LustreError: 50365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 1550.624565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.549558] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1565.164390] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1729.966610] Lustre: server umount lustre-OST0000 complete [ 1746.591632] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1750.040964] Key type lgssc unregistered [ 1750.383312] LNet: 52046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.391884] LNetError: 52046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.462914] LNet: Removed LNI 192.168.202.132@tcp [ 1752.859532] Key type .llcrypt unregistered [ 1752.861232] Key type ._llcrypt unregistered [ 1780.821395] Key type ._llcrypt registered [ 1780.825892] Key type .llcrypt registered [ 1781.098903] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1782.391264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.632614] alg: No test for adler32 (adler32-zlib) [ 1784.659518] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1785.078189] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1786.880523] Key type lgssc registered [ 1788.657734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.802193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.831754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.289716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.847787] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1808.869178] LustreError: 53458:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1808.883194] LustreError: 53458:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1813.986689] LustreError: 53459:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.561216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.207860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.692406] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1830.378617] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1837.110887] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1845.586527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1845.972571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.743827] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1863.696814] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.218565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.218743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1869.237652] Lustre: Skipped 1 previous similar message [ 1872.471951] Lustre: server umount lustre-OST0000 complete [ 1876.450269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.458613] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.480488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.523281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.525603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.641769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.644589] Lustre: Skipped 1 previous similar message [ 1888.738393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1888.742370] Lustre: Skipped 1 previous similar message [ 1890.272149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1890.469967] Lustre: server umount lustre-MDT0000 complete [ 1893.828910] LustreError: 53439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825423 with bad export cookie 328235135003844001 [ 1893.846631] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1894.058184] Lustre: server umount lustre-MDT0001 complete [ 1902.913633] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1905.606896] Key type lgssc unregistered [ 1905.869844] LNet: 56995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.874959] LNetError: 56995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.886270] LNet: Removed LNI 192.168.202.132@tcp [ 1906.580512] Key type .llcrypt unregistered [ 1906.582079] Key type ._llcrypt unregistered [ 1930.970873] Key type ._llcrypt registered [ 1930.973678] Key type .llcrypt registered [ 1931.066711] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1931.929829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.948822] alg: No test for adler32 (adler32-zlib) [ 1933.021716] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 1933.275537] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1934.944346] Key type lgssc registered [ 1936.190378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1944.847823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1944.865708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.352533] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1965.472541] LustreError: 58381:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 1992.160189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.765864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2006.169569] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.107792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.701032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.796731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2018.418903] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2018.805313] LustreError: 58406:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.830034] LustreError: 58406:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2023.910580] LustreError: 58407:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.985688] LustreError: 59418:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.016971] LustreError: 59418:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2027.116218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.535565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2031.344072] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2038.694453] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2045.438544] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2047.977690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.980087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.995594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.016537] Lustre: Skipped 1 previous similar message [ 2053.096868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2053.107769] Lustre: Skipped 1 previous similar message [ 2053.908680] Lustre: server umount lustre-OST0000 complete [ 2078.502806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.508074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.525897] Lustre: Skipped 1 previous similar message [ 2078.531640] Lustre: Skipped 1 previous similar message [ 2082.813511] Lustre: server umount lustre-MDT0000 complete [ 2083.814163] LustreError: 59421:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.847780] LustreError: 59421:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2086.916074] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825616 with bad export cookie 16395362879730976808 [ 2086.927505] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2086.928810] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.356537] Lustre: server umount lustre-MDT0001 complete [ 2105.030861] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2108.088253] Key type lgssc unregistered [ 2108.342916] LNet: 62140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.348800] LNetError: 62140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.377956] LNet: Removed LNI 192.168.202.132@tcp [ 2109.140636] Key type .llcrypt unregistered [ 2109.142520] Key type ._llcrypt unregistered [ 2133.298490] Key type ._llcrypt registered [ 2133.300794] Key type .llcrypt registered [ 2133.426241] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2135.089939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.112174] alg: No test for adler32 (adler32-zlib) [ 2136.175578] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2136.490818] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2138.152265] Key type lgssc registered [ 2139.218206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2147.627345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2147.664789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.300167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.981885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2159.587086] LustreError: 63555:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.616642] LustreError: 63555:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2163.817552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.393397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.507565] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2176.883072] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2182.675739] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.331265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2189.713883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.872960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2196.873574] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2204.534579] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2224.609077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2224.625624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2224.649205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.980512] Lustre: server umount lustre-OST0000 complete [ 2229.729870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2229.737652] Lustre: Skipped 1 previous similar message [ 2229.740120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2229.742784] Lustre: Skipped 1 previous similar message [ 2234.855640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2234.858661] Lustre: Skipped 1 previous similar message [ 2237.662443] Lustre: server umount lustre-MDT0000 complete [ 2239.975635] LustreError: 63556:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2239.992535] LustreError: 63556:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2241.790038] LustreError: 63535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825771 with bad export cookie 17974457630700936092 [ 2241.800448] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2242.132133] Lustre: server umount lustre-MDT0001 complete [ 2253.406569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2256.622204] Key type lgssc unregistered [ 2256.925686] LNet: 67116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.941435] LNetError: 67116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.022356] LNet: Removed LNI 192.168.202.132@tcp [ 2259.144808] Key type .llcrypt unregistered [ 2259.148695] Key type ._llcrypt unregistered [ 2288.448409] Key type ._llcrypt registered [ 2288.452452] Key type .llcrypt registered [ 2288.618288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2289.930061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2290.057830] alg: No test for adler32 (adler32-zlib) [ 2291.190960] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2291.560383] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2293.232177] Key type lgssc registered [ 2294.708882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2303.965499] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 06:17:12 (1787825832) [ 2310.423519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2310.440518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2312.219060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.021267] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2322.413421] LustreError: 68821:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.434861] LustreError: 68821:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2327.523272] LustreError: 68820:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.758801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.225700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.810590] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2342.121186] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2348.312941] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2355.501916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2355.915165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.002303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2362.118836] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2369.832381] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2378.598805] Lustre: Failing over lustre-MDT0000 [ 2378.727187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.737880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.746375] Lustre: Skipped 2 previous similar messages [ 2378.767559] Lustre: Skipped 1 previous similar message [ 2378.900296] Lustre: server umount lustre-MDT0000 complete [ 2381.885675] LustreError: 70504:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.948594] LustreError: 68815:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.970807] LustreError: 68815:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2388.617035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.722719] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.830261] Lustre: *** cfs_fail_loc=135, val=0*** [ 2388.937709] LustreError: 71909:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2388.941834] LustreError: 71909:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -2 [ 2388.946874] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2388.946874] [ 2388.953724] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2388.965698] LustreError: 71897:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2388.983753] LustreError: 71897:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2389.001893] LustreError: 71897:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2389.008057] LustreError: 71897:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2389.044935] Lustre: server umount lustre-MDT0000 complete [ 2389.046639] LustreError: 71897:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2392.077368] LustreError: 68817:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.092719] LustreError: 68817:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2396.065719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.193981] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2396.201837] LustreError: 68812:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2396.214976] LustreError: 72254:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.132@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000ba885cb8) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2396.508273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.581302] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2397.190543] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2401.325635] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2401.814759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2401.847043] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2401.912237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2412.006972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.008160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2412.029416] Lustre: Skipped 1 previous similar message [ 2412.063083] Lustre: Skipped 1 previous similar message [ 2415.010800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2416.064114] Lustre: server umount lustre-OST0000 complete [ 2420.195544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.222114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.234407] Lustre: Skipped 2 previous similar messages [ 2425.313677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2425.322744] Lustre: Skipped 1 previous similar message [ 2426.099730] Lustre: server umount lustre-MDT0000 complete [ 2430.132047] LustreError: 68801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787825959 with bad export cookie 15213073453339937709 [ 2430.142290] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.422377] Lustre: server umount lustre-MDT0001 complete [ 2441.376928] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2444.546420] Key type lgssc unregistered [ 2444.920395] LNet: 73693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.936519] LNetError: 73693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.961329] LNet: Removed LNI 192.168.202.132@tcp [ 2445.927210] Key type .llcrypt unregistered [ 2445.932919] Key type ._llcrypt unregistered [ 2471.360327] Key type ._llcrypt registered [ 2471.364355] Key type .llcrypt registered [ 2471.522604] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2472.770091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2472.785506] alg: No test for adler32 (adler32-zlib) [ 2473.930336] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2474.319459] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2476.088157] Key type lgssc registered [ 2476.932727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.488651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.506327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.943214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2492.274613] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2497.513324] LustreError: 75111:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.537967] LustreError: 75111:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2502.630176] LustreError: 75112:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.731586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.347463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2509.508432] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2518.549243] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2524.482926] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2531.545950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2532.101761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.180394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2540.318374] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2547.292918] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2552.716793] Lustre: Failing over lustre-MDT0000 [ 2552.998241] Lustre: server umount lustre-MDT0000 complete [ 2553.830608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.842045] LustreError: 75106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.846794] Lustre: Skipped 2 previous similar messages [ 2553.876022] LustreError: 75106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2558.946459] LustreError: 76044:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.963446] LustreError: 76044:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2561.641759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.792461] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2562.004485] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2562.131087] LustreError: 78197:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2562.134534] LustreError: 78197:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -9 [ 2562.138825] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2562.138825] [ 2562.147913] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2562.171216] LustreError: 78186:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2562.182281] LustreError: 78186:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2562.202067] LustreError: 78186:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2562.204861] LustreError: 78186:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2562.236335] Lustre: server umount lustre-MDT0000 complete [ 2562.244351] LustreError: 78186:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2564.067085] LustreError: 75106:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.104839] LustreError: 75106:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 2568.975300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.134159] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.192764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2569.418321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.472123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2569.741230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2574.809610] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2574.823986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2574.867197] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2574.936753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2584.545203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2584.556645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.582962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2584.589296] Lustre: Skipped 2 previous similar messages [ 2588.919528] Lustre: server umount lustre-OST0000 complete [ 2595.304433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.308730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.326499] Lustre: Skipped 2 previous similar messages [ 2595.326821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2595.333746] Lustre: Skipped 3 previous similar messages [ 2599.439213] Lustre: server umount lustre-MDT0000 complete [ 2600.418957] LustreError: 75108:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.434093] LustreError: 75108:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2603.276406] LustreError: 75093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826132 with bad export cookie 4432258122585059540 [ 2603.293487] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.781701] Lustre: server umount lustre-MDT0001 complete [ 2616.772856] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2620.258971] Key type lgssc unregistered [ 2620.552757] LNet: 79982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.560947] LNetError: 79982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.594682] LNet: Removed LNI 192.168.202.132@tcp [ 2621.534441] Key type .llcrypt unregistered [ 2621.537212] Key type ._llcrypt unregistered [ 2634.828431] Key type ._llcrypt registered [ 2634.830713] Key type .llcrypt registered [ 2634.936749] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2635.258279] Key type .llcrypt unregistered [ 2635.259796] Key type ._llcrypt unregistered [ 2660.038661] Key type ._llcrypt registered [ 2660.042766] Key type .llcrypt registered [ 2660.212290] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2661.620424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2661.722183] alg: No test for adler32 (adler32-zlib) [ 2662.817688] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2663.034823] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2664.680312] Key type lgssc registered [ 2666.226166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2674.312181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2674.339578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.880029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.061332] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2686.440559] LustreError: 81890:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.460282] LustreError: 81890:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2691.287948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.561107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2691.939208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2697.657118] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2707.296426] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2713.132909] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2721.655148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.243259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2729.736960] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2738.318397] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2746.804507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2762.223174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.226957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.259968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.281293] Lustre: Skipped 1 previous similar message [ 2766.198836] Lustre: server umount lustre-OST0000 complete [ 2772.451653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.468569] Lustre: Skipped 2 previous similar messages [ 2772.470913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.489600] Lustre: Skipped 1 previous similar message [ 2776.447424] Lustre: server umount lustre-MDT0000 complete [ 2777.570170] LustreError: 81891:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.604246] LustreError: 81891:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2780.272717] LustreError: 81872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826309 with bad export cookie 5088729167387958167 [ 2780.280525] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.285080] LustreError: 81872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2780.613767] Lustre: server umount lustre-MDT0001 complete [ 2793.125250] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2795.941570] Key type lgssc unregistered [ 2796.430908] LNet: 85449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2796.443849] LNetError: 85449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2796.477908] LNet: Removed LNI 192.168.202.132@tcp [ 2798.117901] Key type .llcrypt unregistered [ 2798.121127] Key type ._llcrypt unregistered [ 2828.463891] Key type ._llcrypt registered [ 2828.466510] Key type .llcrypt registered [ 2828.576072] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2830.322207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2830.350030] alg: No test for adler32 (adler32-zlib) [ 2831.584497] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2831.928219] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2833.680647] Key type lgssc registered [ 2835.027789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2843.360887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2843.380913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.885615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.934730] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2855.400376] LustreError: 86864:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.411516] LustreError: 86864:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2856.985324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.340534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.010231] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2869.204583] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2875.289718] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2881.232751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2881.497547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2888.275981] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2890.755404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2895.356246] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2904.044081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2904.059077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2904.078354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.520581] Lustre: server umount lustre-OST0000 complete [ 2908.645085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.662068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.697269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.206059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.207256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.323697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.326301] Lustre: Skipped 1 previous similar message [ 2918.466828] Lustre: server umount lustre-MDT0000 complete [ 2921.449829] LustreError: 87868:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.475226] LustreError: 87868:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2922.626742] LustreError: 86844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826451 with bad export cookie 1108159885405766540 [ 2922.636529] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2922.649822] LustreError: 86844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2923.304767] Lustre: server umount lustre-MDT0001 complete [ 2935.826938] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2938.860791] Key type lgssc unregistered [ 2939.174830] LNet: 90402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2939.185932] LNetError: 90402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2940.198977] LNet: Removed LNI 192.168.202.132@tcp [ 2941.139305] Key type .llcrypt unregistered [ 2941.142196] Key type ._llcrypt unregistered [ 2966.286604] Key type ._llcrypt registered [ 2966.287958] Key type .llcrypt registered [ 2966.446839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2967.379777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2967.460270] alg: No test for adler32 (adler32-zlib) [ 2968.673378] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 2969.080571] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2970.787851] Key type lgssc registered [ 2972.312711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2981.258956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2981.271203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.996693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.947320] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2993.125041] LustreError: 91818:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.162901] LustreError: 91818:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2995.102884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.410573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2999.883541] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3010.219887] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3018.041089] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3025.365520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.847706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3027.893480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3032.406330] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3040.649458] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3055.586756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.592045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3055.600735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3055.614426] Lustre: Skipped 1 previous similar message [ 3060.704901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.721351] Lustre: Skipped 2 previous similar messages [ 3065.827752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.840341] Lustre: Skipped 1 previous similar message [ 3068.392757] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3068.537929] Lustre: server umount lustre-OST0000 complete [ 3076.074162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.081239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.089351] Lustre: Skipped 1 previous similar message [ 3076.105664] Lustre: Skipped 1 previous similar message [ 3078.964197] Lustre: server umount lustre-MDT0000 complete [ 3081.188082] LustreError: 91818:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.207598] LustreError: 91818:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3083.072239] LustreError: 91798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826612 with bad export cookie 9777139679313776689 [ 3083.076786] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3083.079274] LustreError: 91798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3083.381548] Lustre: server umount lustre-MDT0001 complete [ 3094.494621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3097.465763] Key type lgssc unregistered [ 3097.831844] LNet: 95378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3097.836141] LNetError: 95378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3097.854617] LNet: Removed LNI 192.168.202.132@tcp [ 3099.010705] Key type .llcrypt unregistered [ 3099.018194] Key type ._llcrypt unregistered [ 3123.211908] Key type ._llcrypt registered [ 3123.214639] Key type .llcrypt registered [ 3123.323360] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3124.577763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3124.628273] alg: No test for adler32 (adler32-zlib) [ 3125.650068] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3125.903186] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3127.688164] Key type lgssc registered [ 3129.116638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3137.717534] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3137.753308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.376866] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3158.496321] LustreError: 96764:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 3185.184398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.304345] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3206.476721] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3213.316294] Lustre: server umount lustre-OST0000 complete [ 3222.054564] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 06:32:30 (1787826750) [ 3228.225938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3228.493530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.763446] LustreError: 98404:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.800837] LustreError: 98404:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3234.319764] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3237.860129] LustreError: 98405:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.988370] LustreError: 98404:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.047928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.331396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3251.248965] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3259.869966] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3266.281083] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.615303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.007662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.932511] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3288.775722] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3298.626797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3314.168842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.181223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.190237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.168495] Lustre: server umount lustre-OST0000 complete [ 3321.319383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3321.334048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.350888] Lustre: Skipped 1 previous similar message [ 3321.354960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.363467] Lustre: Skipped 1 previous similar message [ 3324.393996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.400315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.421464] Lustre: Skipped 1 previous similar message [ 3327.252629] Lustre: server umount lustre-MDT0000 complete [ 3329.505130] LustreError: 98400:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3329.526580] LustreError: 98400:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3331.482571] LustreError: 101168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787826860 with bad export cookie 6291042838311814359 [ 3331.502489] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.800289] Lustre: server umount lustre-MDT0001 complete [ 3343.611839] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3346.989844] Key type lgssc unregistered [ 3347.306407] LNet: 102125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3347.319142] LNetError: 102125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3347.342407] LNet: Removed LNI 192.168.202.132@tcp [ 3348.416477] Key type .llcrypt unregistered [ 3348.420303] Key type ._llcrypt unregistered [ 3394.623118] Key type ._llcrypt registered [ 3394.628227] Key type .llcrypt registered [ 3394.775042] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3396.063279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3396.145921] alg: No test for adler32 (adler32-zlib) [ 3397.308721] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3397.627695] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3399.353046] Key type lgssc registered [ 3400.709801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3409.233209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.256946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.870081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.268895] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3421.156272] LustreError: 104312:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.177091] LustreError: 104312:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 3425.756322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.331478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3431.510864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3439.387805] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3446.513466] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3454.298282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3454.830613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.862838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3460.862346] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3467.753645] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3480.039701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3480.048398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3480.074291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.581756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.586892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.604835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.513682] Lustre: server umount lustre-OST0000 complete [ 3491.812189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.829362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3491.834812] Lustre: Skipped 2 previous similar messages [ 3495.642893] Lustre: server umount lustre-MDT0000 complete [ 3496.931629] LustreError: 104312:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.949808] LustreError: 104312:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 3500.037675] LustreError: 104294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827029 with bad export cookie 12168068641599691234 [ 3500.042736] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3500.050910] LustreError: 104294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3500.488593] Lustre: server umount lustre-MDT0001 complete [ 3511.711659] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3514.454208] Key type lgssc unregistered [ 3514.646979] LNet: 107850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3514.655917] LNetError: 107850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3514.680381] LNet: Removed LNI 192.168.202.132@tcp [ 3515.472834] Key type .llcrypt unregistered [ 3515.475283] Key type ._llcrypt unregistered [ 3533.926388] Key type ._llcrypt registered [ 3533.928866] Key type .llcrypt registered [ 3534.147454] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3536.281293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3536.487934] alg: No test for adler32 (adler32-zlib) [ 3537.709190] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 3538.308649] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3540.154768] Key type lgssc registered [ 3542.010566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3550.729674] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3550.751830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.304444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3571.427194] LustreError: 109096:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 3598.112456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.718217] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3613.159988] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3620.334501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.729634] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3620.751931] LustreError: 110122:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3620.760786] LustreError: 110122:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3620.768626] LustreError: 110122:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3620.789613] LustreError: 109102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827150 with bad export cookie 4502086408847120383 [ 3620.822137] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3620.950298] Lustre: server umount lustre-MDT0000 complete [ 3620.953868] LustreError: 110122:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3648.137226] Lustre: server umount lustre-OST0000 complete [ 3659.478287] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3671.819504] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3711.671904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.020332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3712.065530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3712.274727] Lustre: lustre-MDT0000: new disk, initializing [ 3712.441537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.581312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3718.493279] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3731.186095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.314435] Lustre: 112914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3731.437887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3731.443803] Lustre: Skipped 1 previous similar message [ 3731.545484] Lustre: lustre-MDT0001: new disk, initializing [ 3731.697785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3731.753861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3731.773126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3737.914713] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3748.582594] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3755.069902] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3763.321693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3763.747583] Lustre: lustre-OST0000: new disk, initializing [ 3763.750778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3763.762832] Lustre: 114885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3763.977893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.019089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3771.044936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3771.182600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3771.623889] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3783.058549] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3789.209959] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3789.527933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3796.556699] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3796.953412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3802.083780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.103983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3802.106188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.121685] Lustre: Skipped 1 previous similar message [ 3805.988975] Lustre: server umount lustre-OST0000 complete [ 3812.328490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3812.332447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3812.342123] Lustre: Skipped 2 previous similar messages [ 3812.357048] Lustre: Skipped 1 previous similar message [ 3817.443163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.450689] Lustre: Skipped 1 previous similar message [ 3818.222679] Lustre: server umount lustre-MDT0000 complete [ 3822.795086] LustreError: 112907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827352 with bad export cookie 4502086408847120488 [ 3822.799349] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3822.805901] LustreError: 112907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3823.187499] Lustre: server umount lustre-MDT0001 complete [ 3834.284871] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 06:42:41 (1787827361) [ 3838.596396] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3858.833617] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3871.423972] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3912.445706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.812853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3912.869610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3912.973890] Lustre: lustre-MDT0000: new disk, initializing [ 3913.110401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.131727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3918.710219] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3930.677846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.941071] Lustre: 119790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3931.107503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3931.119248] Lustre: Skipped 1 previous similar message [ 3931.298338] Lustre: lustre-MDT0001: new disk, initializing [ 3931.446875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3931.640621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3931.667104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3937.248894] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3947.223132] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3954.749770] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.289641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.942796] Lustre: lustre-OST0000: new disk, initializing [ 3964.950237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.969327] Lustre: 121759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3965.113787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.508780] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3975.297692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3975.302276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3975.457258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.987222] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3986.868614] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3987.370036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3993.995144] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3994.240636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4001.251504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4001.252581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.262722] LustreError: Skipped 1 previous similar message [ 4001.270445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4001.281032] Lustre: Skipped 1 previous similar message [ 4001.316512] Lustre: Skipped 1 previous similar message [ 4003.465205] Lustre: server umount lustre-OST0000 complete [ 4008.420381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4008.429890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.451930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.081527] Lustre: server umount lustre-MDT0000 complete [ 4016.610323] LustreError: 120790:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4016.633806] LustreError: 120790:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4018.239453] LustreError: 119781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827547 with bad export cookie 4502086408847121370 [ 4018.255310] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.261288] LustreError: 119781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4018.681349] Lustre: server umount lustre-MDT0001 complete [ 4029.849505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.475607] LustreError: 123977:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.637564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.073417] LustreError: 123978:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.270992] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4040.164694] LustreError: 123977:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.293054] LustreError: 123978:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.511067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.218566] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4060.588813] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4067.184155] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.813190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4084.263661] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4093.045595] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4111.849801] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4118.498072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4118.503662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4118.513261] Lustre: Skipped 1 previous similar message [ 4118.517370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4118.525323] Lustre: Skipped 2 previous similar messages [ 4123.630834] Lustre: server umount lustre-OST0000 complete [ 4128.739822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4128.758086] LustreError: Skipped 1 previous similar message [ 4128.772338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.792738] Lustre: Skipped 1 previous similar message [ 4134.478929] Lustre: server umount lustre-MDT0000 complete [ 4136.933157] LustreError: 124705:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.963741] LustreError: 124705:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4139.317433] LustreError: 123957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827668 with bad export cookie 4502086408847122252 [ 4139.338612] LustreError: 123957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4139.353100] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4139.895195] Lustre: server umount lustre-MDT0001 complete [ 4152.539262] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4155.555498] Key type lgssc unregistered [ 4156.002789] LNet: 127634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.013894] LNetError: 127634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4156.033681] LNet: Removed LNI 192.168.202.132@tcp [ 4157.212952] Key type .llcrypt unregistered [ 4157.216693] Key type ._llcrypt unregistered [ 4178.898338] Key type ._llcrypt registered [ 4178.902802] Key type .llcrypt registered [ 4179.026664] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 4198.304979] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4199.844728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4199.943473] alg: No test for adler32 (adler32-zlib) [ 4201.201624] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 4201.672117] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4203.392854] Key type lgssc registered [ 4204.586635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4245.490108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4245.540651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.880770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4246.950105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4247.083734] Lustre: lustre-MDT0000: new disk, initializing [ 4247.201564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.234866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4253.236512] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4264.300136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.476462] Lustre: 130790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4264.617764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4264.635418] Lustre: Skipped 1 previous similar message [ 4264.752861] Lustre: lustre-MDT0001: new disk, initializing [ 4264.903952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4264.981948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.012676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4271.133694] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4284.120334] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4290.300755] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4298.128348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4298.712217] Lustre: lustre-OST0000: new disk, initializing [ 4298.723955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4298.749485] Lustre: 132757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4298.899863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4305.417400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4306.008600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4306.033098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4306.165785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4315.701258] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4321.253888] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4321.514565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4327.397378] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4327.662497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.008145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.011434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.039171] Lustre: Skipped 1 previous similar message [ 4332.091540] Lustre: Skipped 1 previous similar message [ 4337.139353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4337.154800] Lustre: Skipped 1 previous similar message [ 4342.245121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.261062] Lustre: Skipped 1 previous similar message [ 4344.289214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4344.547819] Lustre: server umount lustre-OST0000 complete [ 4351.970990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4351.975705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.007096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4355.904187] Lustre: server umount lustre-MDT0000 complete [ 4360.408439] LustreError: 130783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827889 with bad export cookie 10662083348079822322 [ 4360.418031] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4360.426397] LustreError: 130783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4360.866855] Lustre: server umount lustre-MDT0001 complete [ 4371.430994] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 06:51:39 (1787827899) [ 4379.634245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.289372] LustreError: 135173:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.317556] LustreError: 135173:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 4380.426437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4385.782650] LustreError: 135174:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.037864] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4389.928746] LustreError: 135173:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.979785] LustreError: 135174:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.093956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.747076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4403.566616] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4413.027898] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4420.366834] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4423.138808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4423.144589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.155071] Lustre: Skipped 1 previous similar message [ 4423.160132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.167754] Lustre: Skipped 2 previous similar messages [ 4428.263533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.339789] Lustre: server umount lustre-MDT0000 complete [ 4433.380368] LustreError: 135901:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.404063] LustreError: 135901:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 4433.792767] LustreError: 135155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787827963 with bad export cookie 10662083348079823211 [ 4433.815606] LustreError: 135155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4433.828728] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.303164] Lustre: server umount lustre-MDT0001 complete [ 4445.445978] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 06:52:53 (1787827973) [ 4455.047067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4470.690180] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4475.812990] LustreError: 137593:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 4501.600263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.145995] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4521.574941] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4524.579408] Lustre: server umount lustre-OST0000 complete [ 4535.225896] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 06:54:22 (1787828062) [ 4542.903122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.663479] LustreError: 139041:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.766989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.852462] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4558.303474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.504641] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4571.691344] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4578.285937] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.907806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.555512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.570644] Lustre: Skipped 1 previous similar message [ 4596.109311] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4605.879103] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.137322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4615.152845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.169216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4615.171759] Lustre: Skipped 2 previous similar messages [ 4620.557935] Lustre: server umount lustre-MDT0000 complete [ 4623.332377] LustreError: 140971:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.350282] LustreError: 140971:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 4625.036862] LustreError: 139023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828154 with bad export cookie 10662083348079823876 [ 4625.045882] LustreError: 139023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4625.047499] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4625.497331] Lustre: server umount lustre-MDT0001 complete [ 4640.314526] Lustre: server umount lustre-OST0000 complete [ 4651.254792] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 06:56:19 (1787828179) [ 4659.078239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4659.802655] LustreError: 142698:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4659.887434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.477869] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4675.180710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.564096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4675.573563] Lustre: Skipped 3 previous similar messages [ 4681.299708] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4690.935346] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.869236] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.913235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4712.564657] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4721.032700] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4726.424903] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4729.913297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4731.096125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4735.979522] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4736.356038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4740.075541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4740.083773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.087765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4740.092018] Lustre: Skipped 3 previous similar messages [ 4744.841043] Lustre: server umount lustre-OST0000 complete [ 4756.080852] Lustre: server umount lustre-MDT0000 complete [ 4760.404442] LustreError: 142679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828289 with bad export cookie 10662083348079824947 [ 4760.410708] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4760.418316] LustreError: 142679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4761.060859] Lustre: server umount lustre-MDT0001 complete [ 4770.724889] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 06:58:18 (1787828298) [ 4778.179837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.888705] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4799.009060] LustreError: 146889:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 4824.672580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.681017] Lustre: Skipped 2 previous similar messages [ 4833.053341] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4842.637028] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.284347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.844823] LustreError: 146913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.865518] LustreError: 146913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 4851.041555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4856.652179] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4865.833475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.678732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4879.823638] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4886.315742] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4891.509948] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4891.738499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4896.761744] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4897.002731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4901.865856] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.877694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.881905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.888050] Lustre: Skipped 4 previous similar messages [ 4901.897296] Lustre: Skipped 6 previous similar messages [ 4905.495980] Lustre: server umount lustre-MDT0000 complete [ 4909.300465] LustreError: 146896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828438 with bad export cookie 10662083348079825969 [ 4909.300984] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4909.308217] LustreError: 146896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4909.905341] Lustre: server umount lustre-MDT0001 complete [ 4924.529110] Lustre: server umount lustre-OST0000 complete [ 4934.610236] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:01:02 (1787828462) [ 4943.367773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4959.008652] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4964.129198] LustreError: 151094:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 4997.068593] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5005.602604] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5012.604800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.179317] LustreError: 151119:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.208489] LustreError: 151119:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [ 5013.338260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5018.303649] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5027.885516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.175456] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5042.935518] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5049.231902] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5057.164844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5057.306160] Lustre: lustre-OST0001: new disk, initializing [ 5057.309973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5057.315707] Lustre: 153969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5063.256131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5063.272715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5063.432798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5063.658969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5073.996760] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5079.200564] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5079.430159] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5085.096655] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5085.448239] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5089.263987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.277677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.278932] Lustre: Skipped 1 previous similar message [ 5089.304739] Lustre: Skipped 2 previous similar messages [ 5093.665687] Lustre: server umount lustre-OST0000 complete [ 5099.495936] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5103.850274] Lustre: server umount lustre-OST0001 complete [ 5113.981129] Lustre: server umount lustre-MDT0000 complete [ 5118.045370] LustreError: 153968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828647 with bad export cookie 10662083348079826921 [ 5118.048077] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5118.055272] LustreError: 153968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5118.419626] Lustre: server umount lustre-MDT0001 complete [ 5145.192593] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:04:33 (1787828673) [ 5146.880411] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5148.475655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5150.267878] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:04:38 (1787828678) [ 5157.377641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.595371] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5157.654275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5157.856891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.862674] Lustre: Skipped 6 previous similar messages [ 5162.885835] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5172.163472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5172.236988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5172.254955] Lustre: 157531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5178.775524] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5189.707224] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5195.696898] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.749522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.191566] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5211.810568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5212.780797] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5222.346956] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5227.875884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5228.163883] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5233.660946] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5233.861396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5237.217916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5237.232482] LustreError: Skipped 1 previous similar message [ 5237.240275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5237.257569] Lustre: Skipped 4 previous similar messages [ 5242.606935] Lustre: server umount lustre-OST0000 complete [ 5265.307250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5273.590711] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5277.999611] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.014938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.015054] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5278.036120] Lustre: Skipped 1 previous similar message [ 5282.154901] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5288.515375] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5288.731379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5293.755755] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5293.976031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5301.004666] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d2d88122000.ost_server_uuid 50 [ 5302.776038] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d2d88122000.ost_server_uuid in FULL state after 0 sec [ 5302.945537] LustreError: 161633:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.32@tcp) returned error from blocking AST (req@00000000db9d335b x1874673250091264 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000043fbeebe/0x93f754509fe7e143 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.32@tcp remote: 0x402f8ccca39a3581 expref: 5 pid: 160049 timeout: 5402 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5302.989591] LustreError: lustre-MDT0000: A client on nid 192.168.202.32@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5303.003665] LustreError: 157526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.202.32@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000043fbeebe/0x93f754509fe7e143 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.32@tcp remote: 0x402f8ccca39a3581 expref: 6 pid: 160049 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5308.916103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5308.933591] LustreError: Skipped 1 previous similar message [ 5314.710131] Lustre: server umount lustre-OST0000 complete [ 5325.053331] Lustre: server umount lustre-MDT0000 complete [ 5329.388992] LustreError: 157541:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.405985] LustreError: 157541:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 10 previous similar messages [ 5329.432458] LustreError: 158282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787828858 with bad export cookie 10662083348079828104 [ 5329.459432] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5335.929535] Lustre: server umount lustre-MDT0001 complete [ 5350.326679] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5354.019888] Key type lgssc unregistered [ 5354.322560] LNet: 163648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5354.333442] LNetError: 163648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5354.351054] LNet: Removed LNI 192.168.202.132@tcp [ 5355.464672] Key type .llcrypt unregistered [ 5355.468387] Key type ._llcrypt unregistered [ 5381.990136] Key type ._llcrypt registered [ 5381.995338] Key type .llcrypt registered [ 5382.132996] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5383.825908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5383.995978] alg: No test for adler32 (adler32-zlib) [ 5385.408134] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5385.749770] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5387.448885] Key type lgssc registered [ 5388.829562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5397.289751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5397.312073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.919795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.104510] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5409.257869] LustreError: 165064:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.269343] LustreError: 165064:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5410.282418] LustreError: 165063:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.514917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.080736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5419.045166] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5428.731476] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.674222] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5445.040964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5445.627343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.316519] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5462.558680] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5470.208834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5479.492981] Lustre: Failing over lustre-MDT0000 [ 5479.844918] Lustre: server umount lustre-MDT0000 complete [ 5480.417557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5480.424928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.426749] LustreError: 165058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.435448] Lustre: Skipped 2 previous similar messages [ 5480.461521] LustreError: 165058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5485.542740] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5485.583665] LustreError: 165060:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5490.658199] LustreError: 165064:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.688914] LustreError: 165064:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5492.739457] LustreError: 165044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.32@tcp arrived at 1787829022 with bad export cookie 16299036687331778642 [ 5492.775709] LustreError: 165044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5495.778291] Lustre: 164374:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829009/real 1787829009] req@00000000d3e080e0 x1874674492407040/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787829025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5495.848063] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5500.899948] LustreError: 165058:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.929051] LustreError: 165058:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 5502.089950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.363740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.477112] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5511.660298] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5511.669400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5512.061218] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5514.588983] Lustre: lustre-MDT0000: Denying connection for new client c364e0ec-8a54-42af-873a-ce03818b38a0 (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5525.481172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.486522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.503369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5527.011824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5527.014798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5529.547639] Lustre: server umount lustre-OST0000 complete [ 5532.960265] LustreError: 168817:0:(ldlm_lib.c:3039:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5532.967866] Lustre: 168167:0:(ldlm_lib.c:2439:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5532.972826] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5533.001668] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5533.014134] LustreError: 168167:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000226afefd x1874674492427392/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5533.027853] LustreError: 168167:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5533.036119] LustreError: 168167:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5533.047143] Lustre: lustre-MDT0000: Recovery over after 0:22, of 2 clients 0 recovered and 2 were evicted. [ 5533.476285] Lustre: server umount lustre-MDT0000 complete [ 5536.665971] LustreError: 165044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829065 with bad export cookie 16299036687331778971 [ 5536.670373] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.676025] LustreError: 165044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5536.679943] LustreError: 165791:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.716962] LustreError: 165791:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 5543.115858] Lustre: server umount lustre-MDT0001 complete [ 5554.394890] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5557.096220] Key type lgssc unregistered [ 5557.413843] LNet: 169574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5557.420178] LNetError: 169574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5558.438416] LNet: Removed LNI 192.168.202.132@tcp [ 5559.640088] Key type .llcrypt unregistered [ 5559.649488] Key type ._llcrypt unregistered [ 5583.272935] Key type ._llcrypt registered [ 5583.275928] Key type .llcrypt registered [ 5583.387273] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5584.821424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5584.916095] alg: No test for adler32 (adler32-zlib) [ 5586.040604] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5586.378511] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5588.160229] Key type lgssc registered [ 5589.481420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5596.840977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5596.861946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.312651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5602.710857] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5607.908419] LustreError: 170986:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.916476] LustreError: 170986:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5610.606361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.041629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5615.798404] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5623.240985] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5628.298558] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5633.604421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5633.938742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5639.011368] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5643.195878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5645.187650] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5651.444280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5651.452641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5651.494330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5653.481645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5653.483927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5653.506474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.074687] Lustre: server umount lustre-OST0000 complete [ 5662.181593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5662.193105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5662.217820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5662.225333] Lustre: Skipped 1 previous similar message [ 5666.565704] Lustre: server umount lustre-MDT0000 complete [ 5668.833911] LustreError: 170981:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5668.860435] LustreError: 170981:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5669.717559] LustreError: 170966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829199 with bad export cookie 8869812656869688893 [ 5669.719481] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5669.724182] LustreError: 170966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5669.918595] Lustre: server umount lustre-MDT0001 complete [ 5679.927586] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5682.593518] Key type lgssc unregistered [ 5683.059053] LNet: 174522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5683.072203] LNetError: 174522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5683.095922] LNet: Removed LNI 192.168.202.132@tcp [ 5684.194857] Key type .llcrypt unregistered [ 5684.197063] Key type ._llcrypt unregistered [ 5717.966740] Key type ._llcrypt registered [ 5717.978787] Key type .llcrypt registered [ 5718.099625] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5719.074551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5719.164970] alg: No test for adler32 (adler32-zlib) [ 5720.262234] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5720.495036] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5722.104211] Key type lgssc registered [ 5723.361449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5730.411547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5730.441918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.177209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.241825] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5742.570861] LustreError: 176222:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.593336] LustreError: 176222:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5747.310639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.692203] LustreError: 176221:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5748.188470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5753.991514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5762.554098] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5768.324680] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5776.082732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.589270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5777.601368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5783.427493] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5791.177955] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5798.371682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5798.383547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.430250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5800.678493] Lustre: server umount lustre-OST0000 complete [ 5804.513684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5804.517795] LustreError: Skipped 1 previous similar message [ 5804.520928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.529753] Lustre: Skipped 1 previous similar message [ 5804.534604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5804.538970] Lustre: Skipped 1 previous similar message [ 5808.618424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5808.622339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5810.712875] Lustre: server umount lustre-MDT0000 complete [ 5813.730571] LustreError: 177161:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5813.742597] LustreError: 177161:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5814.194239] LustreError: 176202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829343 with bad export cookie 12983978991067239174 [ 5814.228703] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5814.732716] Lustre: server umount lustre-MDT0001 complete [ 5828.114814] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5831.264648] Key type lgssc unregistered [ 5831.598642] LNet: 179758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5831.609206] LNetError: 179758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5832.686537] LNet: Removed LNI 192.168.202.132@tcp [ 5833.878681] Key type .llcrypt unregistered [ 5833.886503] Key type ._llcrypt unregistered [ 5859.930485] Key type ._llcrypt registered [ 5859.937438] Key type .llcrypt registered [ 5860.095487] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5861.671362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5861.713406] alg: No test for adler32 (adler32-zlib) [ 5862.916197] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5863.204637] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5865.008417] Key type lgssc registered [ 5866.097962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5876.629674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5876.658458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.070286] Lustre: *** cfs_fail_loc=135, val=0*** [ 5878.182099] LustreError: 181283:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5878.191112] LustreError: 181283:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -2 [ 5878.207935] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5878.207935] [ 5878.224730] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5878.250513] LustreError: 181244:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5878.260637] LustreError: 181244:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5878.277343] LustreError: 181244:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5878.301444] LustreError: 181244:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5878.449917] Lustre: server umount lustre-MDT0000 complete [ 5878.455925] LustreError: 181244:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5885.693055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.009606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.406475] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5896.185453] LustreError: 181657:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5896.208415] LustreError: 181657:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 5898.586506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.025841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5903.934686] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5912.142504] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5919.286435] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5921.785134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.819176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5923.887823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5923.892041] Lustre: Skipped 1 previous similar message [ 5928.424935] Lustre: server umount lustre-MDT0000 complete [ 5928.930276] LustreError: 182658:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.967295] LustreError: 182658:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 5932.597666] LustreError: 181637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829461 with bad export cookie 9028819478799314209 [ 5932.601344] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.606610] LustreError: 181637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5933.005812] Lustre: server umount lustre-MDT0001 complete [ 5942.735951] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5945.546398] Key type lgssc unregistered [ 5945.893421] LNet: 183977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5945.906617] LNetError: 183977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5946.919955] LNet: Removed LNI 192.168.202.132@tcp [ 5948.365818] Key type .llcrypt unregistered [ 5948.370950] Key type ._llcrypt unregistered [ 5973.744514] Key type ._llcrypt registered [ 5973.748393] Key type .llcrypt registered [ 5973.916192] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5974.369172] Key type .llcrypt unregistered [ 5974.372166] Key type ._llcrypt unregistered [ 5995.408226] Key type ._llcrypt registered [ 5995.410676] Key type .llcrypt registered [ 5995.549758] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5996.812038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5996.968885] alg: No test for adler32 (adler32-zlib) [ 5998.224420] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 5998.439855] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6000.115594] Key type lgssc registered [ 6001.556717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6009.230871] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6009.252640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.864748] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6029.985515] LustreError: 186117:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 6056.608207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6064.585685] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6073.021280] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.809593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6081.386407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.473334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6086.426209] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6086.632332] LustreError: 186143:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.652696] LustreError: 186143:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6090.737500] LustreError: 187157:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6090.764028] LustreError: 187157:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6095.849751] LustreError: 187602:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6095.884299] LustreError: 187602:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6096.378371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.667749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6102.092640] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6110.480972] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6117.930543] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6121.279908] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6135.798194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.816050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6135.825735] Lustre: Skipped 1 previous similar message [ 6139.769710] Lustre: server umount lustre-OST0000 complete [ 6146.025913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.035753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6146.038478] Lustre: Skipped 1 previous similar message [ 6146.054939] Lustre: Skipped 2 previous similar messages [ 6151.138082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.139706] Lustre: Skipped 1 previous similar message [ 6156.259328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.288760] Lustre: Skipped 1 previous similar message [ 6158.306178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6158.530701] Lustre: server umount lustre-MDT0000 complete [ 6161.398619] LustreError: 187162:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.430184] LustreError: 187162:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6163.148374] LustreError: 186123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829692 with bad export cookie 6569293557812804125 [ 6163.167063] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6163.180413] LustreError: 186123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6163.632985] Lustre: server umount lustre-MDT0001 complete [ 6175.604196] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6178.360057] Key type lgssc unregistered [ 6178.793127] LNet: 190089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6178.805473] LNetError: 190089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6178.821763] LNet: Removed LNI 192.168.202.132@tcp [ 6180.199873] Key type .llcrypt unregistered [ 6180.201952] Key type ._llcrypt unregistered [ 6202.854546] Key type ._llcrypt registered [ 6202.855867] Key type .llcrypt registered [ 6202.910267] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6204.169824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6204.227890] alg: No test for adler32 (adler32-zlib) [ 6205.389934] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6205.773588] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6207.544169] Key type lgssc registered [ 6208.959118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6218.298443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6218.386784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.098369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6226.939991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6230.510362] LustreError: 191494:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6230.540688] LustreError: 191494:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6235.620285] LustreError: 191493:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6240.193508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.672463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6245.841571] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6254.376661] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6260.701554] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6268.285047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.591979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.599941] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6284.779929] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6290.116178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6294.826547] Lustre: Failing over lustre-MDT0000 [ 6295.063237] Lustre: server umount lustre-MDT0000 complete [ 6295.529134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.530883] LustreError: 191488:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.549043] Lustre: Skipped 2 previous similar messages [ 6295.575412] LustreError: 191488:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6300.649920] LustreError: 193946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6300.683025] LustreError: 193946:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6305.771227] LustreError: 191489:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.803257] LustreError: 191489:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 6310.884583] Lustre: 190806:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787829824/real 1787829824] req@00000000de506aec x1874675352233728/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787829840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6310.927303] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6314.552984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.002108] LustreError: 191488:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.016227] LustreError: 191488:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 6321.134814] LustreError: 190803:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b7141c4c x1874675352239104/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6321.523751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.616317] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6322.198556] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6326.794037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6326.824856] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6326.900040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6327.325800] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6332.425379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6340.605083] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6351.843681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.853098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.861549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6351.862373] Lustre: Skipped 1 previous similar message [ 6351.870770] Lustre: Skipped 1 previous similar message [ 6352.375813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.398515] Lustre: Skipped 1 previous similar message [ 6357.489206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6357.502679] Lustre: Skipped 1 previous similar message [ 6362.597294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6366.176300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6366.362696] Lustre: server umount lustre-OST0000 complete [ 6372.835088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6372.840232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.840843] LustreError: Skipped 1 previous similar message [ 6372.856943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6372.856951] Lustre: Skipped 1 previous similar message [ 6376.976902] Lustre: server umount lustre-MDT0000 complete [ 6377.959877] LustreError: 191493:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.996925] LustreError: 191493:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 6381.662613] LustreError: 191476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787829910 with bad export cookie 3392701466117442975 [ 6381.668233] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.669047] LustreError: 191476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6382.103620] Lustre: server umount lustre-MDT0001 complete [ 6393.639912] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6396.759568] Key type lgssc unregistered [ 6397.059247] LNet: 196392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6397.067447] LNetError: 196392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6397.080608] LNet: Removed LNI 192.168.202.132@tcp [ 6398.095115] Key type .llcrypt unregistered [ 6398.100400] Key type ._llcrypt unregistered [ 6421.350288] Key type ._llcrypt registered [ 6421.351853] Key type .llcrypt registered [ 6421.417389] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6422.412500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6422.594893] alg: No test for adler32 (adler32-zlib) [ 6423.705847] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6423.936694] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6425.635519] Key type lgssc registered [ 6426.996390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6434.984380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6435.004806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.375584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.117448] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6446.565645] LustreError: 197804:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6446.581503] LustreError: 197804:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6449.397049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6449.761738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.339252] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6461.558679] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6468.475236] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6477.118762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6477.678586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.757023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6485.705812] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6494.514960] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6499.258422] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6501.686440] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6517.860585] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6530.029994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6530.038478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6530.039635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.048348] Lustre: Skipped 1 previous similar message [ 6534.059441] Lustre: server umount lustre-OST0000 complete [ 6540.262088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6540.272221] Lustre: Skipped 1 previous similar message [ 6540.280313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6540.293516] Lustre: Skipped 1 previous similar message [ 6544.292240] Lustre: server umount lustre-MDT0000 complete [ 6545.380572] LustreError: 198532:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.401603] LustreError: 198532:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6548.904432] LustreError: 197786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830078 with bad export cookie 196301306227269615 [ 6548.923040] LustreError: 197786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6548.923705] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.185265] Lustre: server umount lustre-MDT0001 complete [ 6562.035924] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6565.916907] Key type lgssc unregistered [ 6566.588420] LNet: 201673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6566.614732] LNetError: 201673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6567.671289] LNet: Removed LNI 192.168.202.132@tcp [ 6569.156174] Key type .llcrypt unregistered [ 6569.162220] Key type ._llcrypt unregistered [ 6598.157828] Key type ._llcrypt registered [ 6598.162458] Key type .llcrypt registered [ 6598.328089] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6600.071033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6600.369593] alg: No test for adler32 (adler32-zlib) [ 6601.391108] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6601.624333] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6603.344183] Key type lgssc registered [ 6604.966811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6613.429667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6613.442111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.132244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.073740] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6625.263506] LustreError: 202946:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.289590] LustreError: 202946:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6630.379066] LustreError: 202947:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.976589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.403024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6636.945770] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6646.320726] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6653.479555] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6661.294655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.804126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.050969] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6677.145104] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.381155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6694.023613] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6706.503558] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6715.581183] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6723.324424] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6731.747154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6731.751370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.776608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6731.788385] Lustre: Skipped 1 previous similar message [ 6736.688692] Lustre: server umount lustre-OST0000 complete [ 6741.991769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.995106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6742.027359] Lustre: Skipped 1 previous similar message [ 6742.045757] Lustre: Skipped 2 previous similar messages [ 6747.109631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.128088] Lustre: Skipped 1 previous similar message [ 6747.551822] Lustre: server umount lustre-MDT0000 complete [ 6751.304101] LustreError: 202927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830280 with bad export cookie 11110197723421661126 [ 6751.311958] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6751.317922] LustreError: 202927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6751.586545] Lustre: server umount lustre-MDT0001 complete [ 6762.431221] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6765.373532] Key type lgssc unregistered [ 6765.669807] LNet: 207848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6765.673973] LNetError: 207848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6766.697292] LNet: Removed LNI 192.168.202.132@tcp [ 6767.699256] Key type .llcrypt unregistered [ 6767.702217] Key type ._llcrypt unregistered [ 6791.608243] Key type ._llcrypt registered [ 6791.610925] Key type .llcrypt registered [ 6791.768394] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6793.189603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6793.243645] alg: No test for adler32 (adler32-zlib) [ 6794.447213] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6794.828482] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6796.561807] Key type lgssc registered [ 6798.077962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6805.830873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6805.851905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.196327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6811.554745] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6816.741380] LustreError: 209262:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.755892] LustreError: 209262:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6820.504557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6820.837210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6827.021000] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6836.369725] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.204814] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.491465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.878617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.903299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6857.313842] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6866.655991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.881930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.890716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.901175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.909694] Lustre: Skipped 1 previous similar message [ 6889.196632] Lustre: server umount lustre-OST0000 complete [ 6897.124276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.126421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.132785] Lustre: Skipped 1 previous similar message [ 6897.154833] Lustre: Skipped 2 previous similar messages [ 6902.244528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.361742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.370504] Lustre: Skipped 2 previous similar messages [ 6908.384169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6908.609891] Lustre: server umount lustre-MDT0000 complete [ 6912.490710] LustreError: 211163:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6912.521616] LustreError: 211163:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 6912.741698] LustreError: 209242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830442 with bad export cookie 1822474492461494426 [ 6912.769190] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6913.172050] Lustre: server umount lustre-MDT0001 complete [ 6925.126614] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6928.370263] Key type lgssc unregistered [ 6928.725206] LNet: 212949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6928.736504] LNetError: 212949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6928.770556] LNet: Removed LNI 192.168.202.132@tcp [ 6930.073408] Key type .llcrypt unregistered [ 6930.082480] Key type ._llcrypt unregistered [ 6950.530096] Key type ._llcrypt registered [ 6950.534283] Key type .llcrypt registered [ 6950.688567] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6952.201235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6952.338019] alg: No test for adler32 (adler32-zlib) [ 6953.519565] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 6953.783937] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6955.496384] Key type lgssc registered [ 6956.760146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6965.035284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.051296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.464861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.201776] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6975.973572] LustreError: 214215:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.995658] LustreError: 214215:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 6979.900021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6980.388189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6985.301403] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6991.787833] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6997.048598] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7003.259877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.625513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.490347] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7010.804352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7017.030409] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7031.265368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7031.267138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7031.280448] LustreError: Skipped 1 previous similar message [ 7031.285669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7031.295324] Lustre: Skipped 1 previous similar message [ 7031.334322] Lustre: Skipped 1 previous similar message [ 7036.445214] Lustre: server umount lustre-OST0000 complete [ 7041.511646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.518322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.526299] Lustre: Skipped 1 previous similar message [ 7041.551576] Lustre: Skipped 1 previous similar message [ 7046.632060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7046.756215] Lustre: server umount lustre-MDT0000 complete [ 7050.706461] LustreError: 214197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830580 with bad export cookie 11177013885466856076 [ 7050.717123] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.723252] LustreError: 214197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7051.080262] Lustre: server umount lustre-MDT0001 complete [ 7063.906870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7067.152264] Key type lgssc unregistered [ 7067.465572] LNet: 217903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7067.471771] LNetError: 217903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7067.501655] LNet: Removed LNI 192.168.202.132@tcp [ 7068.435986] Key type .llcrypt unregistered [ 7068.439331] Key type ._llcrypt unregistered [ 7090.047993] Key type ._llcrypt registered [ 7090.057709] Key type .llcrypt registered [ 7090.219283] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7091.940919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7092.068831] alg: No test for adler32 (adler32-zlib) [ 7093.288101] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 7093.754513] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7095.440214] Key type lgssc registered [ 7097.207302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7106.222288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7106.239766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.755282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.956937] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7118.306535] LustreError: 219173:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.332170] LustreError: 219173:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7121.742516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.173150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7127.087730] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7135.094914] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7141.500700] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7148.000971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.364224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7153.463385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7154.683981] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7163.251246] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7173.783951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7173.857873] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7174.121809] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7181.511632] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7190.578764] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7192.467797] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7204.466509] Lustre: Permanently deactivating lustre-OST0001 [ 7204.482823] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7214.236717] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7219.368124] Lustre: Permanently reactivating lustre-OST0001 [ 7219.392327] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7219.398225] Lustre: Skipped 2 previous similar messages [ 7223.492158] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.505704] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7223.512191] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7223.529369] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7223.575121] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7228.897551] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7228.900281] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7228.909042] LustreError: Skipped 1 previous similar message [ 7228.910750] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7228.934413] Lustre: Skipped 2 previous similar messages [ 7231.975041] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7231.983485] Lustre: Skipped 2 previous similar messages [ 7234.647632] Lustre: server umount lustre-OST0001 complete [ 7237.103295] LustreError: 220995:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.123482] LustreError: 220995:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7240.674226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7240.685864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.701745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7242.214512] LustreError: 222680:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.238584] LustreError: 222680:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7245.317950] Lustre: server umount lustre-OST0000 complete [ 7250.400976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7250.409831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.422569] Lustre: Skipped 1 previous similar message [ 7250.434923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7250.442211] Lustre: Skipped 2 previous similar messages [ 7255.306727] Lustre: server umount lustre-MDT0000 complete [ 7257.576553] LustreError: 219900:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.609680] LustreError: 219900:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7259.882564] LustreError: 219154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830789 with bad export cookie 3135190215739160871 [ 7259.891907] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.894899] LustreError: 219154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7260.360334] Lustre: server umount lustre-MDT0001 complete [ 7273.059555] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7276.641151] Key type lgssc unregistered [ 7276.994720] LNet: 224222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.013126] LNetError: 224222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.046074] LNet: Removed LNI 192.168.202.132@tcp [ 7278.494972] Key type .llcrypt unregistered [ 7278.499428] Key type ._llcrypt unregistered [ 7320.570133] Key type ._llcrypt registered [ 7320.572744] Key type .llcrypt registered [ 7320.640679] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7321.331834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7321.523455] alg: No test for adler32 (adler32-zlib) [ 7322.542693] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 7322.665732] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7324.272257] Key type lgssc registered [ 7324.959605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7332.879462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7332.903397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.167898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7334.196457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7334.377911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.415748] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7348.382303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7348.503235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7348.549304] Lustre: 226428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7348.766929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7354.347383] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7363.793720] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7370.736895] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7377.888663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.204504] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7378.524016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.858920] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7387.397799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7394.600967] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7398.730734] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7405.439535] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7414.400632] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7420.205642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7428.999140] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7446.668915] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7446.674397] Lustre: Skipped 1 previous similar message [ 7481.337387] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7481.344762] Lustre: Skipped 3 previous similar messages [ 7551.065677] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7551.080129] Lustre: Skipped 8 previous similar messages [ 7582.208254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.219421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.252794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.266879] Lustre: Skipped 1 previous similar message [ 7585.811922] Lustre: server umount lustre-OST0000 complete [ 7592.424790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.444296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.446223] Lustre: Skipped 2 previous similar messages [ 7592.458598] Lustre: Skipped 1 previous similar message [ 7597.549223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.551822] Lustre: Skipped 1 previous similar message [ 7602.661939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7602.668787] Lustre: Skipped 1 previous similar message [ 7604.193164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7606.492381] Lustre: server umount lustre-MDT0000 complete [ 7610.984457] LustreError: 226422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831140 with bad export cookie 6192998998582246157 [ 7610.987713] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7611.002170] LustreError: 226422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7611.546696] Lustre: server umount lustre-MDT0001 complete [ 7622.964133] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7625.836953] Key type lgssc unregistered [ 7626.134714] LNet: 231230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7626.147595] LNetError: 231230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7626.176839] LNet: Removed LNI 192.168.202.132@tcp [ 7627.056846] Key type .llcrypt unregistered [ 7627.058930] Key type ._llcrypt unregistered [ 7653.486913] Key type ._llcrypt registered [ 7653.489641] Key type .llcrypt registered [ 7653.598808] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7655.113360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7655.171133] alg: No test for adler32 (adler32-zlib) [ 7656.160748] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 7656.426361] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7658.088162] Key type lgssc registered [ 7659.383802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7669.929258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7669.944903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.392955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7676.907518] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7681.509069] LustreError: 232644:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.544656] LustreError: 232644:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 7686.647891] LustreError: 232645:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.444187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.049211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7693.995909] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7702.299186] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7708.573642] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7716.858515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.408313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.165418] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 7733.182967] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.993504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7761.386245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7761.389851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.412409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7761.421595] Lustre: Skipped 1 previous similar message [ 7761.449486] Lustre: Skipped 1 previous similar message [ 7766.517782] LustreError: 235137:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.568836] LustreError: 235137:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7766.800035] Lustre: server umount lustre-OST0000 complete [ 7776.230792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7776.268469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.307765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.843433] Lustre: server umount lustre-MDT0000 complete [ 7781.861083] LustreError: 232645:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7781.919465] LustreError: 232645:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 7785.786484] LustreError: 235250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831315 with bad export cookie 15439315079988070451 [ 7785.787897] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.792028] LustreError: 235250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7786.476222] Lustre: server umount lustre-MDT0001 complete [ 7800.834514] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 7803.757610] Key type lgssc unregistered [ 7804.129344] LNet: 236362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7804.140556] LNetError: 236362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7805.173826] LNet: Removed LNI 192.168.202.132@tcp [ 7806.583732] Key type .llcrypt unregistered [ 7806.588201] Key type ._llcrypt unregistered [ 7833.648834] Key type ._llcrypt registered [ 7833.651797] Key type .llcrypt registered [ 7833.792716] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 7835.203785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7835.305357] alg: No test for adler32 (adler32-zlib) [ 7836.290854] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 7836.541445] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 7838.240176] Key type lgssc registered [ 7839.462249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8110.275635] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8114.110977] Key type lgssc unregistered [ 8114.451088] LNet: 238390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8114.454660] LNetError: 238390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8114.471551] LNet: Removed LNI 192.168.202.132@tcp [ 8115.409524] Key type .llcrypt unregistered [ 8115.414072] Key type ._llcrypt unregistered [ 8142.666578] Key type ._llcrypt registered [ 8142.669023] Key type .llcrypt registered [ 8142.811949] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 8159.768186] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 8161.948683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8161.981782] alg: No test for adler32 (adler32-zlib) [ 8163.277243] Lustre: Lustre: Build Version: 2.17.57_80_g0325ae4 [ 8163.505873] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 8165.152180] Key type lgssc registered [ 8167.404974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.227178] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7878 sec ========== 07:55:30 (1787831730) [ 8203.456973] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:55:31 (1787831731) === [ 8207.332700] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:55:35 (1787831735) === [ 8257.096752] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 8260.533687] Key type lgssc unregistered [ 8260.889762] LNet: 243633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8260.908691] LNetError: 243633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8260.925294] LNet: Removed LNI 192.168.202.132@tcp [ 8261.928911] Key type .llcrypt unregistered [ 8261.931074] Key type ._llcrypt unregistered