[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501694396 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003145] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007896] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011219] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.022615] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027607] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030062] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.033291] *** VALIDATE proc *** [ 0.034243] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037137] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044560] debug: unmapping init [mem 0xffffffff8bc59000-0xffffffff8bc60fff] [ 0.046163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047584] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048031] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054018] ... event mask: 000000070000000f [ 0.055333] rcu: Hierarchical SRCU implementation. [ 0.057580] smp: Bringing up secondary CPUs ... [ 0.058660] x86: Booting SMP configuration: [ 0.059023] .... node #0, CPUs: #1 #2 #3 [ 0.064112] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138297] node 0 deferred pages initialised in 68ms [ 0.142097] devtmpfs: initialized [ 0.143280] x86/mm: Memory block size: 128MB [ 0.147074] gcov: version magic: 0x41383552 [ 0.151387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.156133] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159437] pinctrl core: initialized pinctrl subsystem [ 0.162228] [ 0.162938] ************************************************************* [ 0.166015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168013] ** ** [ 0.171013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.179013] ** IOMMU data structures, which may compromise security on ** [ 0.181012] ** your system. ** [ 0.183013] ** ** [ 0.186014] ** If you see this message and you are not debugging the ** [ 0.188012] ** kernel, report this immediately to your vendor! ** [ 0.191014] ** ** [ 0.194017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196011] ************************************************************* [ 0.199696] NET: Registered protocol family 16 [ 0.201676] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210173] cpuidle: using governor menu [ 0.211916] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214662] PCI: Using configuration type 1 for base access [ 0.217135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227268] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230036] cryptd: max_cpu_qlen set to 1000 [ 0.232077] ACPI: Added _OSI(Module Device) [ 0.233017] ACPI: Added _OSI(Processor Device) [ 0.235030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242851] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250213] ACPI: Interpreter enabled [ 0.252152] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253011] ACPI: Using IOAPIC for interrupt routing [ 0.255176] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260367] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271246] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.285253] acpiphp: Slot [2] registered [ 0.286231] acpiphp: Slot [5] registered [ 0.288117] acpiphp: Slot [6] registered [ 0.289123] acpiphp: Slot [7] registered [ 0.291118] acpiphp: Slot [8] registered [ 0.292138] acpiphp: Slot [9] registered [ 0.294124] acpiphp: Slot [10] registered [ 0.295143] acpiphp: Slot [3] registered [ 0.297115] acpiphp: Slot [4] registered [ 0.298103] acpiphp: Slot [11] registered [ 0.300094] acpiphp: Slot [12] registered [ 0.301158] acpiphp: Slot [13] registered [ 0.303090] acpiphp: Slot [14] registered [ 0.304103] acpiphp: Slot [15] registered [ 0.306141] acpiphp: Slot [16] registered [ 0.307081] acpiphp: Slot [17] registered [ 0.309125] acpiphp: Slot [18] registered [ 0.310145] acpiphp: Slot [19] registered [ 0.312112] acpiphp: Slot [20] registered [ 0.313109] acpiphp: Slot [21] registered [ 0.315152] acpiphp: Slot [22] registered [ 0.316106] acpiphp: Slot [23] registered [ 0.318219] acpiphp: Slot [24] registered [ 0.320120] acpiphp: Slot [25] registered [ 0.321112] acpiphp: Slot [26] registered [ 0.323086] acpiphp: Slot [27] registered [ 0.324091] acpiphp: Slot [28] registered [ 0.325131] acpiphp: Slot [29] registered [ 0.327129] acpiphp: Slot [30] registered [ 0.329111] acpiphp: Slot [31] registered [ 0.331122] PCI host bridge to bus 0000:00 [ 0.332018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.351000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.355000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380440] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384007] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389908] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407977] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.413017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419798] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.437017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.452015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.459739] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.484026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.515021] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.522015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.562101] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.599018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.613202] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.628022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.647018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.657918] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.665013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.671018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.688020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.703963] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706321] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.709385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.714234] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.719103] iommu: Default domain type: Passthrough [ 0.722482] SCSI subsystem initialized [ 0.724160] ACPI: bus type USB registered [ 0.726130] usbcore: registered new interface driver usbfs [ 0.728146] usbcore: registered new interface driver hub [ 0.731117] usbcore: registered new device driver usb [ 0.733257] pps_core: LinuxPPS API ver. 1 registered [ 0.735016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739085] PTP clock support registered [ 0.742053] EDAC MC: Ver: 3.0.0 [ 0.744166] PCI: Using ACPI for IRQ routing [ 0.747150] NetLabel: Initializing [ 0.749013] NetLabel: domain hash size = 128 [ 0.751013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.753087] NetLabel: unlabeled traffic allowed by default [ 0.755131] vgaarb: loaded [ 0.756299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.760023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765313] clocksource: Switched to clocksource kvm-clock [ 0.871413] VFS: Disk quotas dquot_6.6.0 [ 0.872980] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876168] *** VALIDATE ramfs *** [ 0.877668] *** VALIDATE hugetlbfs *** [ 0.879395] pnp: PnP ACPI init [ 0.881825] pnp: PnP ACPI: found 6 devices [ 0.900250] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.904379] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.906977] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.909960] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.913210] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.916287] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.919487] NET: Registered protocol family 2 [ 0.922124] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.927159] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931535] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938783] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.942559] TCP: Hash tables configured (established 65536 bind 65536) [ 0.945863] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.948368] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951904] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.955220] NET: Registered protocol family 1 [ 0.958159] RPC: Registered named UNIX socket transport module. [ 0.960912] RPC: Registered udp transport module. [ 0.962902] RPC: Registered tcp transport module. [ 0.964901] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.967407] NET: Registered protocol family 44 [ 0.969253] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.971579] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.974027] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.976887] PCI: CLS 0 bytes, default 64 [ 0.979214] Unpacking initramfs... [ 2.399781] debug: unmapping init [mem 0xffff92a37cc54000-0xffff92a37ffbffff] [ 2.404469] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.408908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.410696] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.906929] Initialise system trusted keyrings [ 2.908283] Key type blacklist registered [ 2.909609] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.917538] zbud: loaded [ 2.920072] *** VALIDATE nfs *** [ 2.920994] *** VALIDATE nfs4 *** [ 2.922136] pstore: using deflate compression [ 2.925083] Platform Keyring initialized [ 3.034832] NET: Registered protocol family 38 [ 3.036308] Key type asymmetric registered [ 3.037612] Asymmetric key parser 'x509' registered [ 3.039127] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.041703] io scheduler mq-deadline registered [ 3.043203] io scheduler kyber registered [ 3.044635] io scheduler bfq registered [ 3.046494] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049781] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.052728] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.055392] ACPI: Power Button [PWRF] [ 3.060486] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.067698] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.083310] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.093869] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.111931] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.141773] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.172685] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.177827] Non-volatile memory driver v1.3 [ 3.180177] Linux agpgart interface v0.103 [ 3.211993] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.214824] vda: detected capacity change from 0 to 74760192 [ 3.229633] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.232057] vdb: detected capacity change from 0 to 1073741824 [ 3.244856] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.247580] vdc: detected capacity change from 0 to 2621440000 [ 3.260564] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.263304] vdd: detected capacity change from 0 to 2621440000 [ 3.276189] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.278965] vde: detected capacity change from 0 to 4294967296 [ 3.292578] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.295138] vdf: detected capacity change from 0 to 4294967296 [ 3.302604] libphy: Fixed MDIO Bus: probed [ 3.308983] usbcore: registered new interface driver usbserial_generic [ 3.311572] usbserial: USB Serial support registered for generic [ 3.314255] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.318654] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.320251] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.323205] mousedev: PS/2 mouse device common for all mice [ 3.326539] rtc_cmos 00:05: RTC can wake from S4 [ 3.328989] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.329352] rtc_cmos 00:05: registered as rtc0 [ 3.334125] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.337135] intel_pstate: CPU model not supported [ 3.339890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.345201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.345608] hid: raw HID events driver (C) Jiri Kosina [ 3.350834] usbcore: registered new interface driver usbhid [ 3.353090] usbhid: USB HID core driver [ 3.354759] drop_monitor: Initializing network drop monitor service [ 3.357601] Initializing XFRM netlink socket [ 3.360916] NET: Registered protocol family 10 [ 3.364210] Segment Routing with IPv6 [ 3.365938] NET: Registered protocol family 17 [ 3.368246] mpls_gso: MPLS GSO support [ 3.374295] RAS: Correctable Errors collector initialized. [ 3.376284] AVX version of gcm_enc/dec engaged. [ 3.377833] AES CTR mode by8 optimization enabled [ 3.455706] sched_clock: Marking stable (3455685607, 0)->(4415047745, -959362138) [ 3.459601] registered taskstats version 1 [ 3.461615] Loading compiled-in X.509 certificates [ 3.463947] zswap: loaded using pool lzo/zbud [ 3.490024] Key type big_key registered [ 3.501235] Key type encrypted registered [ 3.503163] ima: No TPM chip found, activating TPM-bypass! [ 3.505609] ima: Allocated hash algorithm: sha1 [ 3.507292] ima: No architecture policies found [ 3.509050] evm: Initialising EVM extended attributes: [ 3.510584] evm: security.selinux [ 3.511894] evm: security.ima [ 3.512893] evm: security.capability [ 3.514208] evm: HMAC attrs: 0x1 [ 3.517514] rtc_cmos 00:05: setting system clock to 2026-08-31 05:16:26 UTC (1788153386) [ 3.525330] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 3.528622] debug: unmapping init [mem 0xffffffff8b982000-0xffffffff8bc58fff] [ 3.539200] Write protecting the kernel read-only data: 28672k [ 3.543035] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 3.546113] debug: unmapping init [mem 0xffffffff8a914000-0xffffffff8a9fffff] [ 3.580160] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.589627] systemd[1]: Detected virtualization kvm. [ 3.591742] systemd[1]: Detected architecture x86-64. [ 3.594103] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.620620] systemd[1]: No hostname configured. [ 3.622450] systemd[1]: Set hostname to . [ 3.624861] random: systemd: uninitialized urandom read (16 bytes read) [ 3.627558] systemd[1]: Initializing machine ID from random generator. [ 3.675415] random: ln: uninitialized urandom read (6 bytes read) [ 3.749743] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752974] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.758576] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.763397] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.312413] device-mapper: uevent: version 1.0.3 [ 4.314374] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.957585] random: fast init done [ 4.963257] virtio_net virtio0 ens2: renamed from eth0 [ 5.033683] scsi host0: ata_piix [ 5.047776] scsi host1: ata_piix [ 5.049537] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.052372] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.399239] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.021592] random: crng init done [ 10.023119] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.409774] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.661199] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.928739] SELinux: Disabled at runtime. [ 11.993937] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.003069] systemd[1]: Detected virtualization kvm. [ 12.005115] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.527349] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.532306] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.538314] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.544344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.547416] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.554842] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.561605] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.656638] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.028317] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.297810] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.459337] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.637533] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.659655] EDAC sbridge: Ver: 1.1.2 [ 15.462895] Key type dns_resolver registered [ 15.811676] NFS: Registering the id_resolver key type [ 15.813941] Key type id_resolver registered [ 15.815867] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 42.558138] libcfs: loading out-of-tree module taints kernel. [ 42.579763] Key type ._llcrypt registered [ 42.581799] Key type .llcrypt registered [ 42.632721] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 53.965339] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 56.235308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.284554] alg: No test for adler32 (adler32-zlib) [ 57.994038] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 58.964443] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 60.768620] Key type lgssc registered [ 62.846434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.555786] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 120.803589] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 131.971947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.017253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.277744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.309750] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.437887] Lustre: lustre-MDT0000: new disk, initializing [ 133.553555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.579035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 137.752490] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.521284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.616633] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 150.678512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 150.683505] Lustre: Skipped 1 previous similar message [ 150.780326] Lustre: lustre-MDT0001: new disk, initializing [ 150.886071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 150.917452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 150.929749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 155.430735] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.048092] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 170.053001] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 170.362623] Lustre: lustre-OST0000: new disk, initializing [ 170.371655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 170.381682] Lustre: 8419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 170.479630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.363586] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.750315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.761341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.867481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.355513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.499887] Lustre: lustre-OST0001: new disk, initializing [ 192.504204] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.516727] Lustre: 9492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.610280] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.110971] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.765130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.776625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.856226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.148440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.374868] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 221.490313] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 225.827078] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 229.535873] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 231.868661] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 235.110995] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 237.000487] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:20:17 EDT 2026 [ 238.095142] hrtimer: interrupt took 3808091 ns [ 253.492885] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 255.472525] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 273.384369] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.385022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.407549] Lustre: Skipped 3 previous similar messages [ 273.416167] Lustre: Skipped 3 previous similar messages [ 277.621942] Lustre: server umount lustre-MDT0000 complete [ 280.745846] LustreError: 9490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153663 with bad export cookie 6159145133363292064 [ 280.754697] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.621058] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.626027] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.631415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.642520] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 283.665016] Lustre: Skipped 1 previous similar message [ 285.675856] LustreError: 10232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.677920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.689838] LustreError: 10232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 285.699969] Lustre: Skipped 2 previous similar messages [ 290.785130] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.785176] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.799228] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 290.810462] Lustre: Skipped 1 previous similar message [ 295.392276] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 295.613069] Lustre: server umount lustre-MDT0001 complete [ 298.990967] Lustre: server umount lustre-OST0000 complete [ 302.421795] Lustre: server umount lustre-OST0001 complete [ 307.867289] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 316.201636] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 349.382202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 349.610712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 349.643366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 349.729285] Lustre: lustre-MDT0000: new disk, initializing [ 349.822612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.862386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 354.176988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 364.185736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.312785] Lustre: 15046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 364.369579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 364.373453] Lustre: Skipped 1 previous similar message [ 364.448357] Lustre: lustre-MDT0001: new disk, initializing [ 364.603767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 364.656390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 364.668057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 369.647720] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 377.575798] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 382.329426] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 387.998776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.272390] Lustre: lustre-OST0000: new disk, initializing [ 388.276450] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.283843] Lustre: 17011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 388.378478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.568055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.581761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 389.698768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 393.892464] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 402.296590] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 405.967565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 406.123992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 409.832597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 410.016488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.203030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.204470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.210015] LustreError: Skipped 1 previous similar message [ 415.215577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.221842] Lustre: Skipped 1 previous similar message [ 417.667442] Lustre: server umount lustre-OST0000 complete [ 420.836356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.848829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.858543] Lustre: Skipped 1 previous similar message [ 425.442637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 425.445108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 425.459564] Lustre: Skipped 2 previous similar messages [ 435.168733] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 435.369262] Lustre: server umount lustre-MDT0000 complete [ 435.684364] LustreError: 15057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.689988] LustreError: 15057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 438.880956] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153821 with bad export cookie 6159145133363293604 [ 438.881535] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 438.892508] LustreError: 15039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 439.289933] Lustre: server umount lustre-MDT0001 complete [ 442.385797] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:23:43 (1788153823) [ 447.820315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.180480] LustreError: 19276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.259306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.286760] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 458.081474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.178254] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 466.497298] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.832728] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 477.914785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 484.178942] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 491.677547] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 504.301843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.312298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.333634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.338090] Lustre: Skipped 2 previous similar messages [ 509.058743] Lustre: server umount lustre-OST0000 complete [ 514.531312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.541931] LustreError: Skipped 1 previous similar message [ 519.355673] Lustre: server umount lustre-MDT0000 complete [ 520.676146] LustreError: 19276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.713091] LustreError: 19276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 523.747363] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153906 with bad export cookie 6159145133363294486 [ 523.750877] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.754897] LustreError: 19258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 524.064026] Lustre: server umount lustre-MDT0001 complete [ 534.452417] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 537.228361] Key type lgssc unregistered [ 537.540082] LNet: 22809:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.551587] LNetError: 22809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.576682] LNet: Removed LNI 192.168.204.107@tcp [ 538.455168] Key type .llcrypt unregistered [ 538.458938] Key type ._llcrypt unregistered [ 560.488503] Key type ._llcrypt registered [ 560.489780] Key type .llcrypt registered [ 560.595502] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 561.514325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 561.530053] alg: No test for adler32 (adler32-zlib) [ 562.588414] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 562.886321] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 564.552313] Key type lgssc registered [ 565.684521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 573.087299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.106947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.478245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 577.645718] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 584.680595] LustreError: 24222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.712387] LustreError: 24222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 584.748892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.094413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 589.518231] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 596.437336] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 602.029084] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.272299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 609.645103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.797504] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 619.897258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 625.111615] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 641.508368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 641.520723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.535150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 642.841682] Lustre: server umount lustre-OST0000 complete [ 646.626033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.636302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.644620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.724543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.731830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.744176] Lustre: Skipped 1 previous similar message [ 655.845720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 655.852203] Lustre: Skipped 1 previous similar message [ 656.928961] Lustre: server umount lustre-MDT0000 complete [ 660.512943] LustreError: 24203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154043 with bad export cookie 7155331827387391691 [ 660.524988] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.532226] LustreError: 24203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 660.933389] Lustre: server umount lustre-MDT0001 complete [ 670.757755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 673.284629] Key type lgssc unregistered [ 673.552474] LNet: 28199:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.564872] LNetError: 28199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.607886] LNet: Removed LNI 192.168.204.107@tcp [ 675.554594] Key type .llcrypt unregistered [ 675.557466] Key type ._llcrypt unregistered [ 701.003263] Key type ._llcrypt registered [ 701.005411] Key type .llcrypt registered [ 701.107979] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 702.192904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.429322] alg: No test for adler32 (adler32-zlib) [ 703.522673] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 703.842395] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 705.593469] Key type lgssc registered [ 706.958226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.731802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.746455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.052273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.990784] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 726.514405] LustreError: 29615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.536132] LustreError: 29615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 728.370291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.627206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.431565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 741.401859] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 746.898667] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 759.165459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.502550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.592418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 765.482559] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 772.974308] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 779.242267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.242305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 779.266315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 779.286110] Lustre: Skipped 1 previous similar message [ 780.681277] Lustre: server umount lustre-OST0000 complete [ 784.355626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.363910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 784.376674] Lustre: Skipped 2 previous similar messages [ 789.475662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 789.479776] Lustre: Skipped 1 previous similar message [ 794.594514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 794.606058] Lustre: Skipped 1 previous similar message [ 798.176799] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 798.373146] Lustre: server umount lustre-MDT0000 complete [ 799.716739] LustreError: 30343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 799.745753] LustreError: 30343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 801.927073] LustreError: 29595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154184 with bad export cookie 17067367356768175808 [ 801.930927] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.939984] LustreError: 29595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 802.498910] Lustre: server umount lustre-MDT0001 complete [ 815.142544] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 817.715855] Key type lgssc unregistered [ 818.174971] LNet: 33493:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.190204] LNetError: 33493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.207990] LNet: Removed LNI 192.168.204.107@tcp [ 819.157598] Key type .llcrypt unregistered [ 819.159255] Key type ._llcrypt unregistered [ 843.503086] Key type ._llcrypt registered [ 843.506982] Key type .llcrypt registered [ 843.597781] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 844.132521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.144086] alg: No test for adler32 (adler32-zlib) [ 845.227957] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 845.414477] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 847.080671] Key type lgssc registered [ 847.956490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.925522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.953309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.776691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.028040] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 868.323884] LustreError: 34907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.362537] LustreError: 34907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 873.441885] LustreError: 34906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 875.254404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.767540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 879.237091] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 886.033097] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 891.704076] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.738967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.264184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.345091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 905.907174] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 914.740559] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 926.179772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.191152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.211428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.251460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 929.264095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.265187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.297495] Lustre: Skipped 1 previous similar message [ 930.870195] Lustre: server umount lustre-OST0000 complete [ 934.370741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.385726] Lustre: Skipped 1 previous similar message [ 934.390427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.491726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 939.495253] Lustre: Skipped 2 previous similar messages [ 940.709399] Lustre: server umount lustre-MDT0000 complete [ 944.609485] LustreError: 35973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.622613] LustreError: 35973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 944.724357] LustreError: 36749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154327 with bad export cookie 15227756190794468057 [ 944.726553] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.738152] LustreError: 36749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 945.185743] Lustre: server umount lustre-MDT0001 complete [ 955.670357] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 958.608452] Key type lgssc unregistered [ 958.946509] LNet: 38440:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.955048] LNetError: 38440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.983292] LNet: Removed LNI 192.168.204.107@tcp [ 959.898713] Key type .llcrypt unregistered [ 959.904737] Key type ._llcrypt unregistered [ 983.951170] Key type ._llcrypt registered [ 983.953249] Key type .llcrypt registered [ 984.097101] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 985.527226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.574256] alg: No test for adler32 (adler32-zlib) [ 986.644881] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 986.927502] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 988.656266] Key type lgssc registered [ 990.007966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 998.971598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.016918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1000.492968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.301757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1010.670438] LustreError: 39868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1010.697160] LustreError: 39868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1012.823217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.306602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.733747] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1026.174310] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1032.306553] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.692271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.046786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.169826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1047.184532] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1054.803902] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1065.954756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1065.974266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.996983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.963577] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1069.027809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.029154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.033950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1070.868138] Lustre: server umount lustre-OST0000 complete [ 1094.626774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1094.630887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1094.646828] Lustre: Skipped 1 previous similar message [ 1094.662451] Lustre: Skipped 2 previous similar messages [ 1098.251377] Lustre: server umount lustre-MDT0000 complete [ 1099.745798] LustreError: 41230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1099.771090] LustreError: 41230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1102.698058] LustreError: 39849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154485 with bad export cookie 4882959723954327804 [ 1102.704853] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1102.711126] LustreError: 39849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.145246] Lustre: server umount lustre-MDT0001 complete [ 1120.865842] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1124.076809] Key type lgssc unregistered [ 1124.370671] LNet: 43586:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.381546] LNetError: 43586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.404618] LNet: Removed LNI 192.168.204.107@tcp [ 1125.497367] Key type .llcrypt unregistered [ 1125.504442] Key type ._llcrypt unregistered [ 1149.399689] Key type ._llcrypt registered [ 1149.401813] Key type .llcrypt registered [ 1149.501197] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1150.379125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.459567] alg: No test for adler32 (adler32-zlib) [ 1151.521340] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1151.807559] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1153.560543] Key type lgssc registered [ 1154.609617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.321394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.337973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.776347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1167.545280] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1173.985886] LustreError: 44999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.005085] LustreError: 44999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1174.634156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.234745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.223568] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1189.781376] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1195.409508] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1202.454977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.829904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.225398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1208.935774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1215.328045] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1225.715507] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.724343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.734451] Lustre: Skipped 1 previous similar message [ 1227.187254] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1227.200575] Lustre: Skipped 2 previous similar messages [ 1230.570521] Lustre: server umount lustre-MDT0000 complete [ 1230.819678] LustreError: 46160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.837224] LustreError: 46160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1232.261560] LustreError: 44995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.542833] LustreError: 44979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154617 with bad export cookie 4642580351569510543 [ 1234.554934] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.560758] LustreError: 44979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1235.937833] LustreError: 44996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.938062] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.976487] Lustre: Skipped 1 previous similar message [ 1235.989143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1239.971596] LustreError: 47186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.971761] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1239.984321] LustreError: 47186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1239.987550] Lustre: Skipped 1 previous similar message [ 1240.880418] Lustre: server umount lustre-MDT0001 complete [ 1269.216676] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1269.396505] Lustre: server umount lustre-OST0000 complete [ 1287.285534] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1290.162943] Key type lgssc unregistered [ 1290.529750] LNet: 48860:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.537343] LNetError: 48860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.560780] LNet: Removed LNI 192.168.204.107@tcp [ 1292.635305] Key type .llcrypt unregistered [ 1292.637424] Key type ._llcrypt unregistered [ 1317.844497] Key type ._llcrypt registered [ 1317.846845] Key type .llcrypt registered [ 1317.975597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1319.104775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1319.242136] alg: No test for adler32 (adler32-zlib) [ 1320.334438] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1320.660037] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1322.420789] Key type lgssc registered [ 1323.555960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.192397] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.212441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1346.784471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1351.904444] LustreError: 50248:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 1378.593928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1383.905147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1390.386154] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1557.414879] Lustre: server umount lustre-OST0000 complete [ 1574.872739] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1577.707765] Key type lgssc unregistered [ 1578.055471] LNet: 51920:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.082424] LNetError: 51920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.136441] LNet: Removed LNI 192.168.204.107@tcp [ 1578.987737] Key type .llcrypt unregistered [ 1578.991862] Key type ._llcrypt unregistered [ 1601.580491] Key type ._llcrypt registered [ 1601.583567] Key type .llcrypt registered [ 1601.747883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1602.708916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1602.748251] alg: No test for adler32 (adler32-zlib) [ 1603.994683] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1604.281199] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1605.936400] Key type lgssc registered [ 1607.268996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.082358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.101402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.396746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.758256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1626.596068] LustreError: 53333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.627347] LustreError: 53333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1629.268520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1629.534576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1634.096477] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1641.644355] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1648.422514] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1655.349036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1655.877755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.000383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1662.666460] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1669.922748] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1674.337871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.341279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1678.568056] Lustre: server umount lustre-OST0000 complete [ 1684.961384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.988343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1685.005320] Lustre: Skipped 2 previous similar messages [ 1688.771991] Lustre: server umount lustre-MDT0000 complete [ 1690.081432] LustreError: 55475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.100086] LustreError: 55475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1692.306903] LustreError: 53313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155075 with bad export cookie 7902141491046754999 [ 1692.321548] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.324234] LustreError: 53313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.659036] Lustre: server umount lustre-MDT0001 complete [ 1705.388374] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1709.163492] Key type lgssc unregistered [ 1709.639027] LNet: 56865:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.649507] LNetError: 56865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.668462] LNet: Removed LNI 192.168.204.107@tcp [ 1710.735191] Key type .llcrypt unregistered [ 1710.742484] Key type ._llcrypt unregistered [ 1735.941586] Key type ._llcrypt registered [ 1735.946425] Key type .llcrypt registered [ 1736.172816] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1737.451327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1737.522905] alg: No test for adler32 (adler32-zlib) [ 1738.583563] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1738.841318] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1740.544856] Key type lgssc registered [ 1742.022659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.962421] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.986688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1766.560995] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1771.680547] LustreError: 58253:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 1798.369426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.692035] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1813.331230] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.261771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1820.818408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.904142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1825.289498] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1826.276822] LustreError: 58278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1826.294584] LustreError: 58278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1831.393809] LustreError: 58279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.201183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.451040] LustreError: 59290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.476695] LustreError: 59290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1833.666652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1838.376322] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1846.448246] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.236972] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1854.443857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.450588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.463785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1854.472055] Lustre: Skipped 1 previous similar message [ 1859.563571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.572424] Lustre: Skipped 1 previous similar message [ 1860.471598] Lustre: server umount lustre-OST0000 complete [ 1884.329745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.342534] Lustre: Skipped 1 previous similar message [ 1884.349514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1884.357766] Lustre: Skipped 1 previous similar message [ 1887.499228] Lustre: server umount lustre-MDT0000 complete [ 1889.772801] LustreError: 60303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.809955] LustreError: 60303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1891.040515] LustreError: 58260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155274 with bad export cookie 14686211761710404708 [ 1891.052150] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.062178] LustreError: 58260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1897.441658] Lustre: server umount lustre-MDT0001 complete [ 1908.211324] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1910.753850] Key type lgssc unregistered [ 1910.990411] LNet: 62000:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.000585] LNetError: 62000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.022294] LNet: Removed LNI 192.168.204.107@tcp [ 1911.776067] Key type .llcrypt unregistered [ 1911.779752] Key type ._llcrypt unregistered [ 1934.403288] Key type ._llcrypt registered [ 1934.405385] Key type .llcrypt registered [ 1934.498026] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1935.404802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.418402] alg: No test for adler32 (adler32-zlib) [ 1936.513578] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1936.773442] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1938.571687] Key type lgssc registered [ 1940.197792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1947.757929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.778512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.356199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1954.333579] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1959.907735] LustreError: 63432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.920678] LustreError: 63432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1963.549233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.298845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.076537] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1976.010639] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1981.097767] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.113774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.495902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.687284] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1994.684918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2002.114429] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2020.323842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.327272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.344586] LustreError: Skipped 1 previous similar message [ 2020.346840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.378411] Lustre: Skipped 1 previous similar message [ 2024.953863] Lustre: server umount lustre-OST0000 complete [ 2030.561543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2030.565672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.569353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2030.579153] Lustre: Skipped 1 previous similar message [ 2030.583211] Lustre: Skipped 1 previous similar message [ 2034.364456] Lustre: server umount lustre-MDT0000 complete [ 2035.681730] LustreError: 64161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.692574] LustreError: 64161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2037.995142] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155420 with bad export cookie 1501216900687908793 [ 2038.001206] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.003012] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2038.229722] Lustre: server umount lustre-MDT0001 complete [ 2047.902560] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2050.532712] Key type lgssc unregistered [ 2050.809530] LNet: 66983:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.820676] LNetError: 66983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.847211] LNet: Removed LNI 192.168.204.107@tcp [ 2051.582390] Key type .llcrypt unregistered [ 2051.590432] Key type ._llcrypt unregistered [ 2074.663827] Key type ._llcrypt registered [ 2074.666510] Key type .llcrypt registered [ 2074.756038] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2075.679336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.722457] alg: No test for adler32 (adler32-zlib) [ 2076.758334] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2076.937375] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2078.568165] Key type lgssc registered [ 2079.752873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2088.346645] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 01:51:08 (1788155468) [ 2094.122182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2094.151776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.624098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2099.649070] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2105.831859] LustreError: 68668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.854594] LustreError: 68668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2107.590361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.892293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2111.888686] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2119.000538] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.150529] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.375035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.711887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.295121] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2140.982119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2142.396025] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2147.601698] Lustre: Failing over lustre-MDT0000 [ 2147.864378] Lustre: server umount lustre-MDT0000 complete [ 2148.834114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2148.843399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.852436] LustreError: 69716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.393185] LustreError: 68668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.400511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.407181] LustreError: 68668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2151.427374] Lustre: Skipped 1 previous similar message [ 2154.388300] LustreError: 71109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.418850] LustreError: 71109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2156.088688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.272292] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.402926] Lustre: *** cfs_fail_loc=135, val=0*** [ 2156.493450] LustreError: 71753:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2156.499797] LustreError: 71753:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -2 [ 2156.508885] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2156.508885] [ 2156.517218] LustreError: MGC192.168.204.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2156.536990] LustreError: 71741:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2156.547318] LustreError: 71741:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 2156.564929] LustreError: 71741:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2156.573895] LustreError: 71741:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2156.620862] Lustre: server umount lustre-MDT0000 complete [ 2156.623322] LustreError: 71741:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2159.495731] LustreError: 68664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.512438] LustreError: 68664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2162.505548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.621290] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2162.867284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.927779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2164.629804] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2167.481484] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2168.304337] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2168.330935] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2168.378880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2178.533415] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.542307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2178.585859] Lustre: Skipped 1 previous similar message [ 2178.607109] Lustre: Skipped 1 previous similar message [ 2180.819614] Lustre: server umount lustre-OST0000 complete [ 2188.787610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.789656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.791940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.811394] Lustre: Skipped 1 previous similar message [ 2190.764530] Lustre: server umount lustre-MDT0000 complete [ 2193.890393] LustreError: 69716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.924103] LustreError: 69716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2194.627894] LustreError: 68648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155577 with bad export cookie 11129551754493539185 [ 2194.659446] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.906948] Lustre: server umount lustre-MDT0001 complete [ 2204.611198] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2206.866044] Key type lgssc unregistered [ 2207.093749] LNet: 73530:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.101106] LNetError: 73530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.117670] LNet: Removed LNI 192.168.204.107@tcp [ 2207.860522] Key type .llcrypt unregistered [ 2207.862969] Key type ._llcrypt unregistered [ 2228.184474] Key type ._llcrypt registered [ 2228.187957] Key type .llcrypt registered [ 2228.300604] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2229.105359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2229.128145] alg: No test for adler32 (adler32-zlib) [ 2230.174548] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2230.409800] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2232.072229] Key type lgssc registered [ 2233.248257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2240.527301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2240.546152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.003669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.495443] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2251.238387] LustreError: 74926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.260526] LustreError: 74926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2253.884665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2254.249389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2258.510395] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2265.623939] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.078199] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2277.080175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2277.618813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2283.001267] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2286.863442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2291.527642] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.932217] Lustre: Failing over lustre-MDT0000 [ 2297.222512] Lustre: server umount lustre-MDT0000 complete [ 2297.313757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.318127] LustreError: 74921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.336439] Lustre: Skipped 2 previous similar messages [ 2297.357501] LustreError: 74921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2298.793895] LustreError: 77416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.818036] LustreError: 77416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2302.440167] LustreError: 74922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.461841] LustreError: 74922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2306.701975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.859879] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2307.021495] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2307.189604] LustreError: 78013:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2307.202965] LustreError: 78013:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -9 [ 2307.219678] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2307.219678] [ 2307.235047] LustreError: MGC192.168.204.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2307.249441] LustreError: 78001:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2307.260718] LustreError: 78001:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -9 [ 2307.270553] LustreError: 78001:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 2307.274944] LustreError: 78001:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2307.300698] Lustre: server umount lustre-MDT0000 complete [ 2307.305602] LustreError: 78001:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2307.561310] LustreError: 74920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.580590] LustreError: 74920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2315.046047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2315.193387] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2315.411924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2315.485935] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2319.237056] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2319.837038] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2320.907224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2320.950562] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2321.006254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2325.987239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.997584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.008089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.025649] Lustre: Skipped 1 previous similar message [ 2326.043128] Lustre: Skipped 1 previous similar message [ 2327.710786] Lustre: server umount lustre-OST0000 complete [ 2331.107573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.125063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.134991] Lustre: Skipped 1 previous similar message [ 2336.226290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.245363] Lustre: Skipped 1 previous similar message [ 2337.219714] Lustre: server umount lustre-MDT0000 complete [ 2341.352039] LustreError: 74926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.373594] LustreError: 74926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2341.393990] LustreError: 74905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155724 with bad export cookie 12531130020968053376 [ 2341.407163] LustreError: 74905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2341.407770] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.980632] Lustre: server umount lustre-MDT0001 complete [ 2352.352784] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2354.781977] Key type lgssc unregistered [ 2355.016747] LNet: 79792:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.027415] LNetError: 79792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.041302] LNet: Removed LNI 192.168.204.107@tcp [ 2355.827612] Key type .llcrypt unregistered [ 2355.829952] Key type ._llcrypt unregistered [ 2369.767456] Key type ._llcrypt registered [ 2369.769063] Key type .llcrypt registered [ 2369.943797] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2370.476440] Key type .llcrypt unregistered [ 2370.486340] Key type ._llcrypt unregistered [ 2394.511229] Key type ._llcrypt registered [ 2394.512407] Key type .llcrypt registered [ 2394.613122] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2395.567987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2395.590169] alg: No test for adler32 (adler32-zlib) [ 2396.672806] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2396.949197] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2398.600179] Key type lgssc registered [ 2399.539662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2407.048370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2407.059911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.379751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.937412] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2418.670214] LustreError: 81696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.685896] LustreError: 81696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2419.883883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.331770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2424.897584] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2432.896195] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.318060] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2444.786288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.148249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.964250] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2454.392466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2458.720447] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2466.794273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.801735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.826622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.866658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2469.882251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.887421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.367699] Lustre: server umount lustre-OST0000 complete [ 2474.978863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.982462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.998163] Lustre: Skipped 1 previous similar message [ 2475.020312] Lustre: Skipped 2 previous similar messages [ 2480.098682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.107948] Lustre: Skipped 1 previous similar message [ 2480.826473] Lustre: server umount lustre-MDT0000 complete [ 2484.698812] LustreError: 81676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155867 with bad export cookie 8674408929731943856 [ 2484.702865] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.709214] LustreError: 81676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2485.179775] Lustre: server umount lustre-MDT0001 complete [ 2495.430159] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2498.364133] Key type lgssc unregistered [ 2498.721700] LNet: 85227:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.733766] LNetError: 85227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.761980] LNet: Removed LNI 192.168.204.107@tcp [ 2500.875077] Key type .llcrypt unregistered [ 2500.878093] Key type ._llcrypt unregistered [ 2524.149702] Key type ._llcrypt registered [ 2524.151305] Key type .llcrypt registered [ 2524.269915] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2525.377347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2525.401981] alg: No test for adler32 (adler32-zlib) [ 2526.493329] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2526.785174] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2528.448163] Key type lgssc registered [ 2529.656151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2537.087194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2537.098356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.634442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.040138] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2548.196037] LustreError: 86642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.219958] LustreError: 86642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2551.541514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.003568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2556.542346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2564.104450] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.286154] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2575.593109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2575.982807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.487569] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2583.159670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2591.246720] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2597.870407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2597.881900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.896025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.882431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2598.891752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.895669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2598.930754] Lustre: Skipped 1 previous similar message [ 2604.006911] LustreError: 88464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.022102] LustreError: 88464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2604.155429] Lustre: server umount lustre-OST0000 complete [ 2609.122465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.124408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.143153] Lustre: Skipped 1 previous similar message [ 2609.150484] Lustre: Skipped 1 previous similar message [ 2614.242676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2614.258868] Lustre: Skipped 1 previous similar message [ 2614.741730] Lustre: server umount lustre-MDT0000 complete [ 2618.721351] LustreError: 88483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156001 with bad export cookie 10461446209833331830 [ 2618.727474] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.749677] LustreError: 88483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2619.331254] Lustre: server umount lustre-MDT0001 complete [ 2629.952988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2632.831413] Key type lgssc unregistered [ 2633.149126] LNet: 90174:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.163874] LNetError: 90174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.190932] LNet: Removed LNI 192.168.204.107@tcp [ 2634.032950] Key type .llcrypt unregistered [ 2634.038198] Key type ._llcrypt unregistered [ 2656.828915] Key type ._llcrypt registered [ 2656.831797] Key type .llcrypt registered [ 2656.940552] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2658.140962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.213555] alg: No test for adler32 (adler32-zlib) [ 2659.439729] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2659.704982] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2661.400214] Key type lgssc registered [ 2662.910308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2671.271857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2671.299707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.806833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.393294] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2682.338942] LustreError: 91589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.368256] LustreError: 91589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2686.538052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.161216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.518954] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2698.526605] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2703.862971] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2710.363615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.907175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.710304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2717.048609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2722.438706] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.103909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2736.119141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.148787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.651940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.660656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.673298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.691498] Lustre: Skipped 1 previous similar message [ 2739.954384] Lustre: server umount lustre-OST0000 complete [ 2747.877315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2747.885425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.317914] Lustre: server umount lustre-MDT0000 complete [ 2752.994181] LustreError: 91589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.000804] LustreError: 91589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2753.972449] LustreError: 91571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156136 with bad export cookie 14552379147923213642 [ 2753.975970] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.984047] LustreError: 91571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2754.249903] Lustre: server umount lustre-MDT0001 complete [ 2765.332920] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2768.635746] Key type lgssc unregistered [ 2768.948962] LNet: 95134:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2768.958366] LNetError: 95134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2768.980185] LNet: Removed LNI 192.168.204.107@tcp [ 2769.990223] Key type .llcrypt unregistered [ 2769.997793] Key type ._llcrypt unregistered [ 2794.554249] Key type ._llcrypt registered [ 2794.558609] Key type .llcrypt registered [ 2794.674812] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2795.567727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2795.596822] alg: No test for adler32 (adler32-zlib) [ 2796.623339] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2796.821065] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2798.496372] Key type lgssc registered [ 2799.538827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2806.924696] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2806.988762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.560409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2827.682395] LustreError: 96522:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 2854.432440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.442194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2866.756648] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2872.551870] Lustre: server umount lustre-OST0000 complete [ 2880.302947] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:04:20 (1788156260) [ 2886.676922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.130204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.362885] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2892.267218] LustreError: 98163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.278672] LustreError: 98163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2897.377943] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.099242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.518107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.416784] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2911.688706] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2918.020548] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2924.637081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2925.058285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.762237] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2932.207249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 2937.749834] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2950.625652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.632948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.657217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.680780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2952.684920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2952.692509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.042800] Lustre: server umount lustre-OST0000 complete [ 2960.871888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.878950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.895067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.903025] Lustre: Skipped 1 previous similar message [ 2962.913076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.915522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.703715] Lustre: server umount lustre-MDT0000 complete [ 2968.033550] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.073748] LustreError: 98162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2968.234967] LustreError: 98142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156351 with bad export cookie 15853445688524427681 [ 2968.249555] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2968.561894] Lustre: server umount lustre-MDT0001 complete [ 2979.962200] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2982.655859] Key type lgssc unregistered [ 2982.928569] LNet: 101856:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2982.932963] LNetError: 101856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2983.974417] LNet: Removed LNI 192.168.204.107@tcp [ 2985.022174] Key type .llcrypt unregistered [ 2985.023968] Key type ._llcrypt unregistered [ 3028.674356] Key type ._llcrypt registered [ 3028.676671] Key type .llcrypt registered [ 3028.812091] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3029.822410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3029.843237] alg: No test for adler32 (adler32-zlib) [ 3030.930974] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3031.202853] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3032.913558] Key type lgssc registered [ 3033.760293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3040.027284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3040.052591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3041.393720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.882555] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3052.008029] LustreError: 104042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.030443] LustreError: 104042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3054.025352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.452531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3058.830749] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3066.431829] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.227467] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3078.677810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3079.048217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3084.488585] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3086.192374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3090.799780] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3097.397212] Lustre: server umount lustre-OST0000 complete [ 3100.642145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.653533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.666318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.669022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.673112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.789888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.799712] Lustre: Skipped 2 previous similar messages [ 3110.969945] Lustre: server umount lustre-MDT0000 complete [ 3111.908966] LustreError: 104042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3111.933486] LustreError: 104042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3114.710248] LustreError: 106618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156497 with bad export cookie 16872533227415944156 [ 3114.717771] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.722782] LustreError: 106618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3124.402391] Lustre: server umount lustre-MDT0001 complete [ 3135.311467] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3138.416411] Key type lgssc unregistered [ 3138.785074] LNet: 107574:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3138.797991] LNetError: 107574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3138.811774] LNet: Removed LNI 192.168.204.107@tcp [ 3139.721149] Key type .llcrypt unregistered [ 3139.730452] Key type ._llcrypt unregistered [ 3156.478241] Key type ._llcrypt registered [ 3156.480806] Key type .llcrypt registered [ 3156.554948] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3157.610991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3157.627570] alg: No test for adler32 (adler32-zlib) [ 3158.634786] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3158.829345] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3160.560172] Key type lgssc registered [ 3161.523096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3170.164933] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3170.191551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.824414] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3190.944371] LustreError: 108809:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 3217.633474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.061707] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3230.692550] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3235.811031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.089550] LustreError: MGC192.168.204.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3236.103546] LustreError: 109834:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3236.112768] LustreError: 109834:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 3236.120088] LustreError: 109834:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 3236.211932] Lustre: server umount lustre-MDT0000 complete [ 3236.215168] LustreError: 109834:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3257.824326] Lustre: 108145:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156624/real 1788156624] req@00000000d7424a47 x1875018033138944/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788156640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3257.854542] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3257.879202] LustreError: 108821:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3278.382645] Lustre: server umount lustre-OST0000 complete [ 3288.543465] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3298.900389] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3327.799534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.064295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3328.104079] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3328.195248] Lustre: lustre-MDT0000: new disk, initializing [ 3328.288878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3328.344912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3332.932673] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3344.250367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.390574] Lustre: 112627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3344.430426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3344.433760] Lustre: Skipped 1 previous similar message [ 3344.495896] Lustre: lustre-MDT0001: new disk, initializing [ 3344.615853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.669310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3344.694199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3349.427112] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3358.789119] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3365.189186] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.140280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.560274] Lustre: lustre-OST0000: new disk, initializing [ 3372.563703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3372.569750] Lustre: 114593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.701420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.518844] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3382.337513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3382.351635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3382.542131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3388.133591] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.719511] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3392.950791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3397.179749] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3397.417055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3403.237902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3403.249099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3403.249431] LustreError: Skipped 1 previous similar message [ 3403.251122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3403.263018] Lustre: Skipped 1 previous similar message [ 3405.234114] Lustre: server umount lustre-OST0000 complete [ 3411.425312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3411.429567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.442200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.448546] Lustre: Skipped 1 previous similar message [ 3413.475635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.483760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.325163] Lustre: server umount lustre-MDT0000 complete [ 3419.229614] LustreError: 112620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156802 with bad export cookie 8158592242203023276 [ 3419.250425] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.278559] LustreError: 112620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3419.649242] Lustre: server umount lustre-MDT0001 complete [ 3429.161826] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:13:29 (1788156809) [ 3432.631553] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3445.589813] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3456.204098] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3491.783700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.066267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3492.101128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3492.185278] Lustre: lustre-MDT0000: new disk, initializing [ 3492.269956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.292448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3497.234164] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3507.176954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.375216] Lustre: 119496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3507.467136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3507.475765] Lustre: Skipped 1 previous similar message [ 3507.585457] Lustre: lustre-MDT0001: new disk, initializing [ 3507.674839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3507.723631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3507.737131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3511.930916] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3522.183686] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3527.886877] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3535.558676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.860246] Lustre: lustre-OST0000: new disk, initializing [ 3535.863878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3535.870642] Lustre: 121464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3535.957512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3543.153535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3543.186851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3543.347447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3543.846917] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3553.676237] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3558.545460] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3558.697193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3563.825538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3564.118190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3569.124863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3569.126874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3569.142151] LustreError: Skipped 1 previous similar message [ 3569.145681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3569.165156] Lustre: Skipped 1 previous similar message [ 3569.182077] Lustre: Skipped 2 previous similar messages [ 3572.889716] Lustre: server umount lustre-OST0000 complete [ 3579.368444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.370594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3579.380776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3579.401330] Lustre: Skipped 1 previous similar message [ 3583.502869] Lustre: server umount lustre-MDT0000 complete [ 3584.481933] LustreError: 121677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.516585] LustreError: 121677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3588.506218] LustreError: 119488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156971 with bad export cookie 8158592242203024151 [ 3588.509858] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3588.516693] LustreError: 119488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3588.802900] Lustre: server umount lustre-MDT0001 complete [ 3598.187309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.808742] LustreError: 123684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3598.928179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3602.627448] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3603.939911] LustreError: 123685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.044646] LustreError: 123684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.154702] LustreError: 123685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.128493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.758091] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3628.254968] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3634.664370] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3641.896668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3650.034509] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3658.232996] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3668.208223] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3674.593248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.602233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.615972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.624083] Lustre: Skipped 1 previous similar message [ 3678.474566] Lustre: server umount lustre-OST0000 complete [ 3689.047717] Lustre: server umount lustre-MDT0000 complete [ 3690.977453] LustreError: 123685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.002800] LustreError: 123685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3692.959171] LustreError: 126564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157075 with bad export cookie 8158592242203025040 [ 3692.974048] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.974365] LustreError: 126564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3693.520627] Lustre: server umount lustre-MDT0001 complete [ 3705.183256] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3708.296722] Key type lgssc unregistered [ 3708.651696] LNet: 127316:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.660339] LNetError: 127316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.689713] LNet: Removed LNI 192.168.204.107@tcp [ 3709.559250] Key type .llcrypt unregistered [ 3709.561915] Key type ._llcrypt unregistered [ 3731.212540] Key type ._llcrypt registered [ 3731.215493] Key type .llcrypt registered [ 3731.393474] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3746.203235] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3747.323830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3747.359340] alg: No test for adler32 (adler32-zlib) [ 3748.796473] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3749.191173] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3750.888230] Key type lgssc registered [ 3752.344146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.532215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3789.574726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.915259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3790.967784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3791.159193] Lustre: lustre-MDT0000: new disk, initializing [ 3791.319615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.342380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3796.978452] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3810.242540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.378947] Lustre: 130467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3810.434934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3810.438528] Lustre: Skipped 1 previous similar message [ 3810.519483] Lustre: lustre-MDT0001: new disk, initializing [ 3810.631213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3810.680671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3810.695406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3816.169411] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3826.552401] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3834.168604] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3842.498312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3843.210155] Lustre: lustre-OST0000: new disk, initializing [ 3843.231624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3843.243520] Lustre: 132436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3843.444540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.377848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3849.388589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3849.644904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3851.058775] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3862.547399] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3869.378918] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3869.692920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3874.778694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3874.937212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3880.495523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3880.522447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.522511] LustreError: Skipped 1 previous similar message [ 3880.526620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3880.554164] Lustre: Skipped 1 previous similar message [ 3883.800360] Lustre: server umount lustre-OST0000 complete [ 3890.659463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.693905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3890.705285] Lustre: Skipped 1 previous similar message [ 3895.853134] Lustre: server umount lustre-MDT0000 complete [ 3900.471583] LustreError: 130458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157283 with bad export cookie 2641588262018920085 [ 3900.476025] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3900.487262] LustreError: 130458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3900.844097] Lustre: server umount lustre-MDT0001 complete [ 3913.663419] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:21:33 (1788157293) [ 3921.426659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.219665] LustreError: 134853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.234124] LustreError: 134853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3922.309436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.531973] LustreError: 134854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.690809] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3932.652149] LustreError: 134853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.761757] LustreError: 134854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.181567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.516259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3943.760985] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3951.706191] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.875328] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.387623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.399308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.417596] Lustre: Skipped 1 previous similar message [ 3964.429725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.245527] Lustre: server umount lustre-MDT0000 complete [ 3969.724262] LustreError: 134835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157352 with bad export cookie 2641588262018920967 [ 3969.738035] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.068941] Lustre: server umount lustre-MDT0001 complete [ 3978.483178] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:22:39 (1788157359) [ 3985.476205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4001.057944] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4006.177165] LustreError: 137268:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 4031.970372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.293208] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4048.094684] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4050.180329] Lustre: server umount lustre-OST0000 complete [ 4059.658931] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:23:59 (1788157439) [ 4067.349028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4067.871333] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4067.934875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.542427] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4077.026904] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.044980] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4082.132121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4087.895947] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4096.292357] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4103.431142] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4111.674900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.122775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.130360] Lustre: Skipped 1 previous similar message [ 4119.589386] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4127.852081] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4137.955914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.965841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.974728] Lustre: Skipped 3 previous similar messages [ 4137.979848] Lustre: Skipped 4 previous similar messages [ 4142.027044] Lustre: server umount lustre-MDT0000 complete [ 4143.076074] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.098528] LustreError: 138714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4145.671416] LustreError: 138697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157528 with bad export cookie 2641588262018921632 [ 4145.677425] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4145.685968] LustreError: 138697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4146.024739] Lustre: server umount lustre-MDT0001 complete [ 4160.237495] Lustre: server umount lustre-OST0000 complete [ 4169.558913] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:25:50 (1788157550) [ 4176.647454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.303649] LustreError: 142369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.477791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.926195] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4191.697487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4197.494847] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4205.494964] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4212.314636] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4219.473473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4222.859085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4226.458261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4233.814798] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4239.168864] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4239.398718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4244.469400] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4244.736618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4247.525404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4247.529879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.536837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.536894] Lustre: Skipped 1 previous similar message [ 4247.597381] Lustre: Skipped 1 previous similar message [ 4252.936093] Lustre: server umount lustre-OST0000 complete [ 4257.763472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.792079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.808106] Lustre: Skipped 4 previous similar messages [ 4265.493320] Lustre: server umount lustre-MDT0000 complete [ 4268.004825] LustreError: 142370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.024242] LustreError: 142370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4269.466700] LustreError: 142349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157652 with bad export cookie 2641588262018922703 [ 4269.475300] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4269.480508] LustreError: 142349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4269.927819] Lustre: server umount lustre-MDT0001 complete [ 4279.342773] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:27:39 (1788157659) [ 4286.408939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.112803] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4307.232820] LustreError: 146529:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 4332.962391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.973506] Lustre: Skipped 2 previous similar messages [ 4339.788995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4347.156358] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4352.771530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.404977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4357.483641] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4366.603336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.436612] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4379.297448] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4384.455930] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4389.517134] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4389.875542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.421387] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4396.754184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4399.080408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.081747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.111581] Lustre: Skipped 3 previous similar messages [ 4399.115129] Lustre: Skipped 3 previous similar messages [ 4405.846395] Lustre: server umount lustre-MDT0000 complete [ 4409.313596] LustreError: 148306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.349881] LustreError: 148306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4409.808731] LustreError: 146537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157792 with bad export cookie 2641588262018923725 [ 4409.819451] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.820926] LustreError: 146537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4410.034155] Lustre: server umount lustre-MDT0001 complete [ 4423.911478] Lustre: server umount lustre-OST0000 complete [ 4433.439926] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:30:13 (1788157813) [ 4440.061617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4455.649803] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4460.768481] LustreError: 150723:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 4492.457444] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4500.232712] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.327122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.859903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4511.003521] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4520.208035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.230716] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4533.938449] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4540.079807] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4546.932352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4547.105165] Lustre: lustre-OST0001: new disk, initializing [ 4547.108951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4547.116904] Lustre: 153597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4553.360405] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4556.317972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4556.338219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4556.500221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4561.586520] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4565.693798] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4565.874855] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4569.855247] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4570.045358] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4572.135509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4572.140789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4572.148270] Lustre: Skipped 1 previous similar message [ 4572.158568] Lustre: Skipped 5 previous similar messages [ 4577.837621] Lustre: server umount lustre-OST0000 complete [ 4582.373647] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.382486] LustreError: Skipped 1 previous similar message [ 4587.395367] Lustre: server umount lustre-OST0001 complete [ 4592.100686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4597.017135] Lustre: server umount lustre-MDT0000 complete [ 4600.772328] LustreError: 150730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157983 with bad export cookie 2641588262018924677 [ 4600.778383] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.785939] LustreError: 150730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4601.112099] Lustre: server umount lustre-MDT0001 complete [ 4625.058928] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:33:25 (1788158005) [ 4627.283163] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4629.253169] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4630.634557] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:33:31 (1788158011) [ 4636.124292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.472251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4636.536659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4636.904317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.909213] Lustre: Skipped 6 previous similar messages [ 4641.339622] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4650.636833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.790297] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4650.835197] Lustre: 157157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4655.707205] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4664.385530] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4670.794680] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4678.163983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4678.398106] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4685.505446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4688.527927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4693.627992] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4698.620961] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4698.848572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4704.042206] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4704.308319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4708.836655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4708.856126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.874088] Lustre: Skipped 4 previous similar messages [ 4708.883416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4708.894233] Lustre: Skipped 6 previous similar messages [ 4712.747743] Lustre: server umount lustre-OST0000 complete [ 4735.086420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4741.986289] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4749.227716] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4749.858817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.881535] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4754.086645] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4754.348677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4759.083190] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.097918] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4759.296701] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4759.489059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4765.547648] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a16c6b4d000.ost_server_uuid 50 [ 4767.016977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a16c6b4d000.ost_server_uuid in FULL state after 0 sec [ 4767.232716] LustreError: 157162:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.7@tcp) returned error from blocking AST (req@00000000181a5a9a x1875018653107200 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000004f1d4611/0x24a8ce87507ac9df lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.7@tcp remote: 0xa543729b486642bf expref: 5 pid: 157163 timeout: 4867 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4767.284101] LustreError: lustre-MDT0000: A client on nid 192.168.204.7@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4767.292066] LustreError: 157152:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.7@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000004f1d4611/0x24a8ce87507ac9df lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.7@tcp remote: 0xa543729b486642bf expref: 6 pid: 157163 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4772.846877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4772.865629] LustreError: Skipped 1 previous similar message [ 4777.953949] Lustre: server umount lustre-OST0000 complete [ 4796.384334] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4796.514838] Lustre: server umount lustre-MDT0000 complete [ 4799.969389] LustreError: 158268:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.982289] LustreError: 158268:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4800.368462] LustreError: 157149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158183 with bad export cookie 2641588262018925860 [ 4800.369625] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4800.376736] LustreError: 157149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4806.795420] Lustre: server umount lustre-MDT0001 complete [ 4819.871894] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4822.805228] Key type lgssc unregistered [ 4823.090864] LNet: 163271:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4823.098647] LNetError: 163271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4823.111673] LNet: Removed LNI 192.168.204.107@tcp [ 4823.868195] Key type .llcrypt unregistered [ 4823.870996] Key type ._llcrypt unregistered [ 4848.176967] Key type ._llcrypt registered [ 4848.180334] Key type .llcrypt registered [ 4848.285451] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4849.160228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4849.213476] alg: No test for adler32 (adler32-zlib) [ 4850.196395] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4850.476410] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4852.169154] Key type lgssc registered [ 4853.249191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4861.647067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4861.681822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.240197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4867.912198] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4873.702101] LustreError: 164685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.718344] LustreError: 164685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4878.825622] LustreError: 164684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.893044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.556946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4884.259436] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4891.505448] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4896.820593] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4903.704165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4904.063530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4908.144876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4909.326576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4916.028406] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4924.046733] Lustre: Failing over lustre-MDT0000 [ 4924.765973] Lustre: server umount lustre-MDT0000 complete [ 4925.932073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4925.948342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.972550] LustreError: 165848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4928.993952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4928.996235] LustreError: 164685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.067727] LustreError: 164685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4934.119173] LustreError: 165906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.153108] LustreError: 165906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4937.358243] LustreError: 164664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1788158320 with bad export cookie 10340243475501451526 [ 4944.288250] Lustre: 163999:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788158311/real 1788158311] req@00000000bd2114f6 x1875019807354496/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788158327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4944.328925] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.340708] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4944.351620] LustreError: 164681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4944.507125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.619852] LustreError: 163995:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bf646974 x1875019807359744/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4955.909133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.963925] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4960.730500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4961.266492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4961.278285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4963.169912] Lustre: lustre-MDT0000: Denying connection for new client b8f2987c-c4c4-4890-8137-ae5805437384 (at 192.168.204.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4968.325958] Lustre: lustre-MDT0000: Denying connection for new client b8f2987c-c4c4-4890-8137-ae5805437384 (at 192.168.204.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 4972.015676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4972.031360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.049059] Lustre: Skipped 1 previous similar message [ 4972.058204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.622798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.623057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.643719] Lustre: Skipped 1 previous similar message [ 4977.895175] Lustre: server umount lustre-OST0000 complete [ 4982.197123] LustreError: 168416:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4982.212832] Lustre: 167765:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4982.226668] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4982.247526] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4982.266220] LustreError: 167765:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000acc978c1 x1875019807375232/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4982.303394] LustreError: 167765:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4982.317065] LustreError: 167765:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4982.326296] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 4982.630152] Lustre: server umount lustre-MDT0000 complete [ 4986.721222] LustreError: 167382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158369 with bad export cookie 10340243475501451624 [ 4986.735454] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.735968] LustreError: 167382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4986.763807] LustreError: 163995:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 4989.280440] Lustre: server umount lustre-MDT0001 complete [ 5000.673847] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5003.504700] Key type lgssc unregistered [ 5003.820889] LNet: 169170:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5003.833180] LNetError: 169170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5004.904439] LNet: Removed LNI 192.168.204.107@tcp [ 5005.784091] Key type .llcrypt unregistered [ 5005.786085] Key type ._llcrypt unregistered [ 5028.345562] Key type ._llcrypt registered [ 5028.347293] Key type .llcrypt registered [ 5028.436214] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5029.752705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5029.780180] alg: No test for adler32 (adler32-zlib) [ 5030.900164] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5031.189824] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5032.888627] Key type lgssc registered [ 5034.202559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5042.112225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5042.127968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.525763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5048.258799] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5053.925720] LustreError: 170583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5053.945854] LustreError: 170583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5056.060197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5056.511915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5061.184765] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5068.742944] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5073.971696] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5080.325443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.691055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.991365] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5088.898056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5094.367626] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5102.562761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5102.582363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.598194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5104.610843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5104.617578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.621779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5106.267539] Lustre: server umount lustre-OST0000 complete [ 5109.731986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.753673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.764429] Lustre: Skipped 2 previous similar messages [ 5114.853047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5114.867226] Lustre: Skipped 1 previous similar message [ 5115.952354] Lustre: server umount lustre-MDT0000 complete [ 5119.970832] LustreError: 170580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.996293] LustreError: 170580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5120.128273] LustreError: 173159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158503 with bad export cookie 8562592790742709058 [ 5120.138318] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.148058] LustreError: 173159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5120.530116] Lustre: server umount lustre-MDT0001 complete [ 5130.418109] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5133.336370] Key type lgssc unregistered [ 5133.642705] LNet: 174114:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5133.652241] LNetError: 174114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5133.672308] LNet: Removed LNI 192.168.204.107@tcp [ 5134.463149] Key type .llcrypt unregistered [ 5134.465461] Key type ._llcrypt unregistered [ 5166.318230] Key type ._llcrypt registered [ 5166.328203] Key type .llcrypt registered [ 5166.437407] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5167.363906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5167.385455] alg: No test for adler32 (adler32-zlib) [ 5168.381231] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5168.590174] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5170.248897] Key type lgssc registered [ 5171.435824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5180.107376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5180.156895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.558035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5185.776530] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5191.161039] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.185948] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5194.791734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.262624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5199.089360] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5205.779197] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5212.227903] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5220.369135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.846092] LustreError: 177641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.190502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.320694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5230.395242] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5238.565428] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5246.270432] Lustre: server umount lustre-OST0000 complete [ 5251.553240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5251.558238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.570191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5252.071717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5252.086211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5252.093854] Lustre: Skipped 1 previous similar message [ 5257.190792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5257.201231] Lustre: Skipped 1 previous similar message [ 5259.347220] Lustre: server umount lustre-MDT0000 complete [ 5262.304518] Lustre: 175131:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788158629/real 1788158629] req@00000000ab0427d4 x1875020140800256/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788158645 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5262.305161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5262.328814] Lustre: 175131:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5262.342126] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.363532] LustreError: 175820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5262.932261] LustreError: 175802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158645 with bad export cookie 17515178436529821772 [ 5262.940847] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5262.943479] LustreError: 175802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5263.376177] Lustre: server umount lustre-MDT0001 complete [ 5274.150512] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5277.103946] Key type lgssc unregistered [ 5277.420779] LNet: 179352:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5277.431211] LNetError: 179352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5277.449379] LNet: Removed LNI 192.168.204.107@tcp [ 5278.461116] Key type .llcrypt unregistered [ 5278.463430] Key type ._llcrypt unregistered [ 5299.577837] Key type ._llcrypt registered [ 5299.586503] Key type .llcrypt registered [ 5299.693117] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5301.033199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5301.085071] alg: No test for adler32 (adler32-zlib) [ 5302.065826] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5302.331677] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5304.032177] Key type lgssc registered [ 5305.077118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5317.060089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5317.122168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.624488] Lustre: *** cfs_fail_loc=135, val=0*** [ 5318.725481] LustreError: 180876:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5318.733780] LustreError: 180876:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.107@tcp: cfg command failed: rc = -2 [ 5318.743124] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5318.743124] [ 5318.760627] LustreError: MGC192.168.204.107@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5318.773851] LustreError: 180837:0:(tgt_mount.c:1951:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5318.789617] LustreError: 180837:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [ 5318.806148] LustreError: 180837:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-MDT0000 [ 5318.814628] LustreError: 180837:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5318.945822] Lustre: server umount lustre-MDT0000 complete [ 5318.955404] LustreError: 180837:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5325.737878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5326.186946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5330.169786] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5336.548234] LustreError: 181246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5336.571479] LustreError: 181246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5338.132692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5338.557849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.952653] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5350.392045] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5356.104381] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5358.308771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.322485] Lustre: Skipped 1 previous similar message [ 5358.323054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5358.333771] Lustre: Skipped 1 previous similar message [ 5363.494189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.503561] Lustre: Skipped 1 previous similar message [ 5364.655734] Lustre: server umount lustre-MDT0000 complete [ 5368.064740] LustreError: 181227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158751 with bad export cookie 4100864582649401452 [ 5368.065438] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5368.083721] LustreError: 181227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5368.336447] Lustre: server umount lustre-MDT0001 complete [ 5379.152440] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5382.053523] Key type lgssc unregistered [ 5382.412646] LNet: 183568:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5382.418191] LNetError: 183568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5382.434083] LNet: Removed LNI 192.168.204.107@tcp [ 5383.171343] Key type .llcrypt unregistered [ 5383.173957] Key type ._llcrypt unregistered [ 5401.031439] Key type ._llcrypt registered [ 5401.034068] Key type .llcrypt registered [ 5401.112556] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5401.363069] Key type .llcrypt unregistered [ 5401.367109] Key type ._llcrypt unregistered [ 5419.054416] Key type ._llcrypt registered [ 5419.057586] Key type .llcrypt registered [ 5419.155571] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5419.566452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5419.601909] alg: No test for adler32 (adler32-zlib) [ 5420.720965] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5421.076085] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5422.881493] Key type lgssc registered [ 5424.453996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5432.328963] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5432.347089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5447.904794] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5453.024352] LustreError: 185702:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 5479.713943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5485.777950] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5493.496891] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5500.675481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5501.313656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5501.372793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5505.570679] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5506.531248] LustreError: 185727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.557947] LustreError: 185727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5511.650205] LustreError: 185728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.232175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5513.667269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5518.325083] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5526.431532] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5531.295756] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5534.228461] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5544.418107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.430323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.448124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5545.451688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.453190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5548.846435] Lustre: server umount lustre-OST0000 complete [ 5554.658497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5554.667168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5554.685788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5554.693966] Lustre: Skipped 1 previous similar message [ 5558.971391] Lustre: server umount lustre-MDT0000 complete [ 5560.804857] LustreError: 186739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5560.833992] LustreError: 186739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5563.083954] LustreError: 185708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158946 with bad export cookie 4437528276807185701 [ 5563.090334] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.094947] LustreError: 185708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5563.511911] Lustre: server umount lustre-MDT0001 complete [ 5575.476458] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5578.505644] Key type lgssc unregistered [ 5578.862545] LNet: 189627:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5578.880227] LNetError: 189627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5578.902180] LNet: Removed LNI 192.168.204.107@tcp [ 5579.873270] Key type .llcrypt unregistered [ 5579.877970] Key type ._llcrypt unregistered [ 5606.150463] Key type ._llcrypt registered [ 5606.153747] Key type .llcrypt registered [ 5606.321541] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5607.664670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5607.705776] alg: No test for adler32 (adler32-zlib) [ 5608.982768] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5609.287453] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5610.976184] Key type lgssc registered [ 5612.620750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5622.022392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5622.050167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.641684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.350423] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5634.018913] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.039886] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5636.074026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.460590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5639.764874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5647.899936] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5653.210171] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5661.231952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5661.779343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5667.810874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5669.947154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5674.663328] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.399585] Lustre: Failing over lustre-MDT0000 [ 5679.734778] Lustre: server umount lustre-MDT0000 complete [ 5680.105539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5680.124682] Lustre: Skipped 2 previous similar messages [ 5680.135215] LustreError: 191036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.159857] LustreError: 191036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5681.186603] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.205632] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5686.246490] LustreError: 191037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.266287] LustreError: 191037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5691.362734] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.384340] LustreError: 191041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5696.417832] Lustre: 190353:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788159063/real 1788159063] req@000000000dfe845c x1875020603218944/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788159079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5696.459466] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.138240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.601429] LustreError: 191769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.619577] LustreError: 191769:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5706.722593] LustreError: 190350:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eba996f0 x1875020603225856/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5706.936445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.982283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5708.035570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5711.580923] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5712.379845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5712.464034] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5712.561474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5717.090694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5725.447408] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5734.882226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5734.889312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.904222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5736.947139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5736.957232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5736.982245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.870766] Lustre: server umount lustre-OST0000 complete [ 5748.199586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.202771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5748.217820] Lustre: Skipped 1 previous similar message [ 5748.237667] Lustre: Skipped 3 previous similar messages [ 5751.237800] Lustre: server umount lustre-MDT0000 complete [ 5753.315389] LustreError: 193342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.344809] LustreError: 193342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5755.507582] LustreError: 191776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159138 with bad export cookie 16679863300871909940 [ 5755.512192] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.526171] LustreError: 191776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5755.974900] Lustre: server umount lustre-MDT0001 complete [ 5768.479748] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5771.277573] Key type lgssc unregistered [ 5771.567717] LNet: 195933:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5771.574230] LNetError: 195933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5771.586688] LNet: Removed LNI 192.168.204.107@tcp [ 5772.444435] Key type .llcrypt unregistered [ 5772.445824] Key type ._llcrypt unregistered [ 5796.732858] Key type ._llcrypt registered [ 5796.738847] Key type .llcrypt registered [ 5796.864599] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5798.251822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5798.380483] alg: No test for adler32 (adler32-zlib) [ 5799.500390] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5799.740818] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5801.416285] Key type lgssc registered [ 5802.690567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5810.522229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5810.558758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.988744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5816.430752] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5821.417724] LustreError: 197346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.453042] LustreError: 197346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5825.826882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.149094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5831.227871] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5838.859338] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5845.633782] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.675816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5853.011181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.147297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5859.308629] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5866.898749] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5870.714981] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5878.681559] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5895.264292] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5902.314351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.315157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5902.315515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5902.329915] Lustre: Skipped 1 previous similar message [ 5907.234617] Lustre: server umount lustre-OST0000 complete [ 5912.549259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.551870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5912.562178] Lustre: Skipped 1 previous similar message [ 5912.567629] Lustre: Skipped 1 previous similar message [ 5917.667411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5917.675957] Lustre: Skipped 2 previous similar messages [ 5922.795826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.802097] Lustre: Skipped 1 previous similar message [ 5925.344273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5925.711913] Lustre: server umount lustre-MDT0000 complete [ 5927.910926] LustreError: 198841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.937121] LustreError: 198841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5929.669229] LustreError: 197328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159312 with bad export cookie 781103466673221563 [ 5929.669785] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5929.681122] LustreError: 197328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5930.010808] Lustre: server umount lustre-MDT0001 complete [ 5941.850735] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 5944.738752] Key type lgssc unregistered [ 5945.090639] LNet: 201218:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5945.097392] LNetError: 201218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5945.118039] LNet: Removed LNI 192.168.204.107@tcp [ 5946.028426] Key type .llcrypt unregistered [ 5946.033521] Key type ._llcrypt unregistered [ 5964.030736] Key type ._llcrypt registered [ 5964.033347] Key type .llcrypt registered [ 5964.121463] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5964.737056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5964.756946] alg: No test for adler32 (adler32-zlib) [ 5965.767590] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5966.026715] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 5967.720380] Key type lgssc registered [ 5968.986730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5976.740461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5976.754499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.288347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5982.823290] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5988.837179] LustreError: 202477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.860076] LustreError: 202477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5992.023842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.481528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5997.596896] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6005.117482] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6011.587328] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6018.268099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6018.664836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.790697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6024.713447] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6032.556422] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6039.002378] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6052.133893] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6058.783534] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6064.169905] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6073.317506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.318056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6073.324785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.333099] Lustre: Skipped 1 previous similar message [ 6078.434898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6078.456211] Lustre: Skipped 1 previous similar message [ 6079.347684] Lustre: server umount lustre-OST0000 complete [ 6083.559351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.588183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6083.600751] Lustre: Skipped 2 previous similar messages [ 6088.676969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6088.684112] Lustre: Skipped 1 previous similar message [ 6089.795398] Lustre: server umount lustre-MDT0000 complete [ 6092.599945] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159475 with bad export cookie 4045390708094931842 [ 6092.603563] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6092.607740] LustreError: 202458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6092.890732] Lustre: server umount lustre-MDT0001 complete [ 6103.861949] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6107.329424] Key type lgssc unregistered [ 6107.679205] LNet: 207353:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6107.704283] LNetError: 207353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.730705] LNet: Removed LNI 192.168.204.107@tcp [ 6108.820520] Key type .llcrypt unregistered [ 6108.823341] Key type ._llcrypt unregistered [ 6134.973359] Key type ._llcrypt registered [ 6134.975636] Key type .llcrypt registered [ 6135.090088] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6135.935914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6135.993914] alg: No test for adler32 (adler32-zlib) [ 6137.184538] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6137.495908] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6139.216202] Key type lgssc registered [ 6140.339384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6146.870264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6146.882855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.477594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6152.535272] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6158.827827] LustreError: 208766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.865233] LustreError: 208766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6162.333913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6162.839420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6167.295920] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6174.259309] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6180.837820] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6189.269779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.549548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.655212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6196.460178] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6203.597692] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6218.219548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.226308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6218.243729] Lustre: Skipped 1 previous similar message [ 6218.251925] Lustre: Skipped 1 previous similar message [ 6218.252052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6223.169119] Lustre: server umount lustre-OST0000 complete [ 6228.452212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.456881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.472155] Lustre: Skipped 1 previous similar message [ 6233.574623] LustreError: 209494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.578395] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6233.584276] LustreError: 209494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6233.670261] Lustre: server umount lustre-MDT0000 complete [ 6237.958791] Lustre: server umount lustre-MDT0001 complete [ 6249.260738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6251.976466] Key type lgssc unregistered [ 6252.303715] LNet: 212452:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6252.314161] LNetError: 212452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6252.339060] LNet: Removed LNI 192.168.204.107@tcp [ 6253.307229] Key type .llcrypt unregistered [ 6253.309302] Key type ._llcrypt unregistered [ 6273.556073] Key type ._llcrypt registered [ 6273.557234] Key type .llcrypt registered [ 6273.667508] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6275.136714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.183835] alg: No test for adler32 (adler32-zlib) [ 6276.193547] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6276.431242] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6278.162080] Key type lgssc registered [ 6279.525215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6287.781589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6287.803433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.289789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6293.470313] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6299.629333] LustreError: 213734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.644566] LustreError: 213734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6301.760581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.242252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6306.731585] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6313.623944] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6318.716772] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6324.963400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6325.316726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.243388] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6334.525454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6337.663966] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6352.868042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.873963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.885816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6354.920181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6354.932716] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6354.942825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.902550] Lustre: server umount lustre-OST0000 complete [ 6363.617787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6363.626412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.657943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6363.665944] Lustre: Skipped 1 previous similar message [ 6367.037995] Lustre: server umount lustre-MDT0000 complete [ 6370.275447] LustreError: 213733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6370.301807] LustreError: 213733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6370.785200] LustreError: 213713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159753 with bad export cookie 4184574014224366664 [ 6370.790794] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6370.797236] LustreError: 213713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6371.142352] Lustre: server umount lustre-MDT0001 complete [ 6382.330751] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6385.234853] Key type lgssc unregistered [ 6385.639772] LNet: 217415:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6385.644852] LNetError: 217415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6386.676905] LNet: Removed LNI 192.168.204.107@tcp [ 6387.560733] Key type .llcrypt unregistered [ 6387.565699] Key type ._llcrypt unregistered [ 6407.877445] Key type ._llcrypt registered [ 6407.883259] Key type .llcrypt registered [ 6407.993320] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6409.282530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6409.382940] alg: No test for adler32 (adler32-zlib) [ 6410.507449] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6410.804961] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6412.545883] Key type lgssc registered [ 6413.629511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6421.214136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6421.229330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6422.713801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.854398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6432.229434] LustreError: 218680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.255483] LustreError: 218680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6434.483775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.881614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6438.776807] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6445.842384] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6451.279959] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6457.567257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.944025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.548463] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6467.203717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6471.779001] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.319557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.528949] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6481.675974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6487.422667] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6496.193561] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6497.736281] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6510.619417] Lustre: Permanently deactivating lustre-OST0001 [ 6510.663106] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6519.837277] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6530.816353] Lustre: Permanently reactivating lustre-OST0001 [ 6530.837839] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6530.844491] Lustre: Skipped 2 previous similar messages [ 6539.362224] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6539.377629] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6539.386760] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6539.395981] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6539.465829] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6542.830818] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.840773] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6542.853030] Lustre: Skipped 2 previous similar messages [ 6542.880091] Lustre: Skipped 1 previous similar message [ 6543.800759] Lustre: server umount lustre-OST0001 complete [ 6547.941985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6547.943896] LustreError: 220501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.946668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6547.974580] Lustre: Skipped 1 previous similar message [ 6548.000470] LustreError: 220501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6553.062468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.064065] LustreError: 220501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6553.073552] Lustre: Skipped 2 previous similar messages [ 6553.961844] Lustre: server umount lustre-OST0000 complete [ 6558.180632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6558.206873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6558.218950] Lustre: Skipped 1 previous similar message [ 6563.299719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6563.312156] Lustre: Skipped 1 previous similar message [ 6564.191726] Lustre: server umount lustre-MDT0000 complete [ 6567.588769] LustreError: 219414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159950 with bad export cookie 13405820730952811700 [ 6567.591760] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6567.603662] LustreError: 219414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6567.852318] Lustre: server umount lustre-MDT0001 complete [ 6581.649766] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6584.888859] Key type lgssc unregistered [ 6585.428494] LNet: 223777:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.439691] LNetError: 223777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.455185] LNet: Removed LNI 192.168.204.107@tcp [ 6586.705672] Key type .llcrypt unregistered [ 6586.711099] Key type ._llcrypt unregistered [ 6630.914317] Key type ._llcrypt registered [ 6630.918988] Key type .llcrypt registered [ 6631.080368] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6632.078061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6632.185086] alg: No test for adler32 (adler32-zlib) [ 6633.277220] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6633.547967] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6635.216512] Key type lgssc registered [ 6636.694712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6645.769305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6645.783307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.088935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6647.148501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6647.374308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6652.421571] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6661.318597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.484832] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6661.532082] Lustre: 225986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6661.805296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6666.416052] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6673.756171] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6679.252739] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.933505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.176499] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6686.441968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.431209] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6695.066065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6699.472421] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6703.065931] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6707.760255] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6712.481713] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6718.430563] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6727.239608] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6748.464637] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6748.470096] Lustre: Skipped 2 previous similar messages [ 6785.931753] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6785.940165] Lustre: Skipped 4 previous similar messages [ 6858.544434] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6858.553168] Lustre: Skipped 7 previous similar messages [ 6879.202214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6879.215749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.229994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6879.715228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6879.730628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.740895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6882.089747] Lustre: server umount lustre-OST0000 complete [ 6887.393292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6887.401660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.425076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6889.955328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6889.955414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.076179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6895.087141] Lustre: Skipped 2 previous similar messages [ 6900.192143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6900.420314] Lustre: server umount lustre-MDT0000 complete [ 6904.625406] LustreError: 228661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160287 with bad export cookie 2243324137538728610 [ 6904.640582] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6904.650120] LustreError: 228661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6905.214280] Lustre: server umount lustre-MDT0001 complete [ 6915.624897] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6919.071241] Key type lgssc unregistered [ 6919.332488] LNet: 230807:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6919.357971] LNetError: 230807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6919.385402] LNet: Removed LNI 192.168.204.107@tcp [ 6920.751635] Key type .llcrypt unregistered [ 6920.754537] Key type ._llcrypt unregistered [ 6947.037396] Key type ._llcrypt registered [ 6947.040853] Key type .llcrypt registered [ 6947.210861] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6948.163457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.212908] alg: No test for adler32 (adler32-zlib) [ 6949.459822] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6949.729395] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6951.448736] Key type lgssc registered [ 6952.875904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6961.471731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6961.490455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.058819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6968.460559] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6972.385466] LustreError: 232222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6972.412902] LustreError: 232222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6977.515260] LustreError: 232223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.065058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.641395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6983.082184] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6990.286920] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.625552] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7003.672110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.973948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7008.085496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7010.716538] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7018.905984] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.827836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7033.832489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7033.841360] LustreError: Skipped 1 previous similar message [ 7033.844113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7033.877293] Lustre: Skipped 1 previous similar message [ 7038.953063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7038.955402] Lustre: Skipped 1 previous similar message [ 7039.599907] Lustre: server umount lustre-OST0000 complete [ 7044.070718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.080773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.089386] Lustre: Skipped 1 previous similar message [ 7044.098428] Lustre: Skipped 2 previous similar messages [ 7049.185902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7049.198646] Lustre: Skipped 1 previous similar message [ 7049.526875] Lustre: server umount lustre-MDT0000 complete [ 7053.385696] LustreError: 232204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160436 with bad export cookie 12803456768003100292 [ 7053.398368] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.402121] LustreError: 232204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7053.763880] Lustre: server umount lustre-MDT0001 complete [ 7063.230766] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7065.642855] Key type lgssc unregistered [ 7065.945719] LNet: 235909:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7065.950682] LNetError: 235909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7065.965799] LNet: Removed LNI 192.168.204.107@tcp [ 7066.773546] Key type .llcrypt unregistered [ 7066.776725] Key type ._llcrypt unregistered [ 7090.193829] Key type ._llcrypt registered [ 7090.196411] Key type .llcrypt registered [ 7090.322794] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7091.348037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7091.366720] alg: No test for adler32 (adler32-zlib) [ 7092.373228] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7092.643819] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7094.352327] Key type lgssc registered [ 7095.238958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.285809] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7371.374235] Key type lgssc unregistered [ 7371.700570] LNet: 237930:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7371.708704] LNetError: 237930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7371.728719] LNet: Removed LNI 192.168.204.107@tcp [ 7372.587573] Key type .llcrypt unregistered [ 7372.594170] Key type ._llcrypt unregistered [ 7399.791381] Key type ._llcrypt registered [ 7399.798290] Key type .llcrypt registered [ 7400.026502] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7419.290572] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7420.657319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7420.760566] alg: No test for adler32 (adler32-zlib) [ 7422.071465] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7422.556536] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7424.288203] Key type lgssc registered [ 7426.099467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7460.486979] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7221 sec ========== 03:20:40 (1788160840) [ 7462.620699] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:20:42 (1788160842) === [ 7466.788892] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:20:46 (1788160846) === [ 7508.096885] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7511.107752] Key type lgssc unregistered [ 7511.420121] LNet: 243171:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7511.434581] LNetError: 243171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7511.450470] LNet: Removed LNI 192.168.204.107@tcp [ 7512.492079] Key type .llcrypt unregistered [ 7512.494074] Key type ._llcrypt unregistered