[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519146626 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003305] x2apic enabled [ 0.004014] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007938] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010164] LSM: Security Framework initializing [ 0.011074] Yama: becoming mindful. [ 0.012064] SELinux: Initializing. [ 0.013094] *** VALIDATE selinux *** [ 0.021700] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027502] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030148] *** VALIDATE tmpfs *** [ 0.032230] *** VALIDATE proc *** [ 0.033261] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035311] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042351] debug: unmapping init [mem 0xffffffffab859000-0xffffffffab860fff] [ 0.044177] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049019] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053319] rcu: Hierarchical SRCU implementation. [ 0.055515] smp: Bringing up secondary CPUs ... [ 0.056689] x86: Booting SMP configuration: [ 0.057029] .... node #0, CPUs: #1 #2 #3 [ 0.061012] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.226497] node 0 deferred pages initialised in 159ms [ 0.230165] devtmpfs: initialized [ 0.231285] x86/mm: Memory block size: 128MB [ 0.235051] gcov: version magic: 0x41383552 [ 0.237428] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.241101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243370] pinctrl core: initialized pinctrl subsystem [ 0.245159] [ 0.245649] ************************************************************* [ 0.248014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251012] ** ** [ 0.253010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255018] ** ** [ 0.257015] ** This means that this kernel is built to expose internal ** [ 0.259013] ** IOMMU data structures, which may compromise security on ** [ 0.262012] ** your system. ** [ 0.264013] ** ** [ 0.266014] ** If you see this message and you are not debugging the ** [ 0.269013] ** kernel, report this immediately to your vendor! ** [ 0.271011] ** ** [ 0.273014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275016] ************************************************************* [ 0.277810] NET: Registered protocol family 16 [ 0.280551] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289027] cpuidle: using governor menu [ 0.290753] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294029] PCI: Using configuration type 1 for base access [ 0.296131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.311227] cryptd: max_cpu_qlen set to 1000 [ 0.315258] ACPI: Added _OSI(Module Device) [ 0.317071] ACPI: Added _OSI(Processor Device) [ 0.317785] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.319010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324160] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329435] ACPI: Interpreter enabled [ 0.331063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333012] ACPI: Using IOAPIC for interrupt routing [ 0.334130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336444] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.349042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.351021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.354078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359487] acpiphp: Slot [2] registered [ 0.361116] acpiphp: Slot [5] registered [ 0.363116] acpiphp: Slot [6] registered [ 0.364124] acpiphp: Slot [7] registered [ 0.365114] acpiphp: Slot [8] registered [ 0.367199] acpiphp: Slot [9] registered [ 0.368096] acpiphp: Slot [10] registered [ 0.369107] acpiphp: Slot [3] registered [ 0.371124] acpiphp: Slot [4] registered [ 0.373069] acpiphp: Slot [11] registered [ 0.374119] acpiphp: Slot [12] registered [ 0.375112] acpiphp: Slot [13] registered [ 0.377126] acpiphp: Slot [14] registered [ 0.378093] acpiphp: Slot [15] registered [ 0.380127] acpiphp: Slot [16] registered [ 0.382100] acpiphp: Slot [17] registered [ 0.383351] acpiphp: Slot [18] registered [ 0.385099] acpiphp: Slot [19] registered [ 0.387108] acpiphp: Slot [20] registered [ 0.388109] acpiphp: Slot [21] registered [ 0.390111] acpiphp: Slot [22] registered [ 0.391086] acpiphp: Slot [23] registered [ 0.393093] acpiphp: Slot [24] registered [ 0.394140] acpiphp: Slot [25] registered [ 0.395126] acpiphp: Slot [26] registered [ 0.397109] acpiphp: Slot [27] registered [ 0.398088] acpiphp: Slot [28] registered [ 0.400086] acpiphp: Slot [29] registered [ 0.401118] acpiphp: Slot [30] registered [ 0.402116] acpiphp: Slot [31] registered [ 0.404082] PCI host bridge to bus 0000:00 [ 0.405017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.408087] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.413021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.418032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427417] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.438018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.443462] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.446019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.448017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.449017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452646] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455705] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.458048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.460858] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.466014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.491414] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.502016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.508016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.527016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.539269] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.548019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.561020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.584018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.597612] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.614017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.632020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.643399] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.650017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.655015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.669019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676248] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.686016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.696018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.724028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.733084] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.740020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.748025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.772020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.782432] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.784382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.788417] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791844] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.798175] iommu: Default domain type: Passthrough [ 0.801131] SCSI subsystem initialized [ 0.802119] ACPI: bus type USB registered [ 0.804185] usbcore: registered new interface driver usbfs [ 0.806077] usbcore: registered new interface driver hub [ 0.808083] usbcore: registered new device driver usb [ 0.810182] pps_core: LinuxPPS API ver. 1 registered [ 0.811010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.815094] PTP clock support registered [ 0.817331] EDAC MC: Ver: 3.0.0 [ 0.820305] PCI: Using ACPI for IRQ routing [ 0.823034] NetLabel: Initializing [ 0.824009] NetLabel: domain hash size = 128 [ 0.826012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.829081] NetLabel: unlabeled traffic allowed by default [ 0.831291] vgaarb: loaded [ 0.833353] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.835015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.843098] clocksource: Switched to clocksource kvm-clock [ 0.981294] VFS: Disk quotas dquot_6.6.0 [ 0.983313] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.985724] *** VALIDATE ramfs *** [ 0.986714] *** VALIDATE hugetlbfs *** [ 0.987833] pnp: PnP ACPI init [ 0.990594] pnp: PnP ACPI: found 6 devices [ 1.007180] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.010445] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.012606] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.015076] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.017625] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.019984] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.022556] NET: Registered protocol family 2 [ 1.025251] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.030467] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.034312] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.039963] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.043763] TCP: Hash tables configured (established 65536 bind 65536) [ 1.047148] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.050453] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.052657] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.055842] NET: Registered protocol family 1 [ 1.059711] RPC: Registered named UNIX socket transport module. [ 1.062648] RPC: Registered udp transport module. [ 1.064652] RPC: Registered tcp transport module. [ 1.066462] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.069029] NET: Registered protocol family 44 [ 1.071014] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.073962] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.076398] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.078592] PCI: CLS 0 bytes, default 64 [ 1.080721] Unpacking initramfs... [ 2.536378] debug: unmapping init [mem 0xffff9f4c3cc54000-0xffff9f4c3ffbffff] [ 2.543092] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.545339] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.549775] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.017677] Initialise system trusted keyrings [ 3.019470] Key type blacklist registered [ 3.021401] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.030613] zbud: loaded [ 3.033625] *** VALIDATE nfs *** [ 3.034989] *** VALIDATE nfs4 *** [ 3.036589] pstore: using deflate compression [ 3.040099] Platform Keyring initialized [ 3.138924] NET: Registered protocol family 38 [ 3.141918] Key type asymmetric registered [ 3.144515] Asymmetric key parser 'x509' registered [ 3.146674] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.149860] io scheduler mq-deadline registered [ 3.151404] io scheduler kyber registered [ 3.153142] io scheduler bfq registered [ 3.155347] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.159142] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161583] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.165278] ACPI: Power Button [PWRF] [ 3.171994] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.179670] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.198275] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.211958] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.230040] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.258116] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.303588] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.309155] Non-volatile memory driver v1.3 [ 3.310791] Linux agpgart interface v0.103 [ 3.345208] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.348165] vda: detected capacity change from 0 to 76685312 [ 3.364581] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.368344] vdb: detected capacity change from 0 to 1073741824 [ 3.386156] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.389147] vdc: detected capacity change from 0 to 2621440000 [ 3.405567] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.408367] vdd: detected capacity change from 0 to 2621440000 [ 3.420359] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.423520] vde: detected capacity change from 0 to 4294967296 [ 3.438399] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.440517] vdf: detected capacity change from 0 to 4294967296 [ 3.446359] libphy: Fixed MDIO Bus: probed [ 3.463702] usbcore: registered new interface driver usbserial_generic [ 3.465803] usbserial: USB Serial support registered for generic [ 3.467872] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.472287] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.474073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.476668] mousedev: PS/2 mouse device common for all mice [ 3.479443] rtc_cmos 00:05: RTC can wake from S4 [ 3.481519] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.483601] rtc_cmos 00:05: registered as rtc0 [ 3.489924] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.489970] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.495067] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.496394] intel_pstate: CPU model not supported [ 3.502576] hid: raw HID events driver (C) Jiri Kosina [ 3.504700] usbcore: registered new interface driver usbhid [ 3.506079] usbhid: USB HID core driver [ 3.507701] drop_monitor: Initializing network drop monitor service [ 3.510069] Initializing XFRM netlink socket [ 3.512087] NET: Registered protocol family 10 [ 3.514744] Segment Routing with IPv6 [ 3.515892] NET: Registered protocol family 17 [ 3.518401] mpls_gso: MPLS GSO support [ 3.525201] RAS: Correctable Errors collector initialized. [ 3.527332] AVX version of gcm_enc/dec engaged. [ 3.528610] AES CTR mode by8 optimization enabled [ 3.608062] sched_clock: Marking stable (3608022895, 0)->(4564058438, -956035543) [ 3.611162] registered taskstats version 1 [ 3.612800] Loading compiled-in X.509 certificates [ 3.614666] zswap: loaded using pool lzo/zbud [ 3.637973] Key type big_key registered [ 3.650801] Key type encrypted registered [ 3.654586] ima: No TPM chip found, activating TPM-bypass! [ 3.660997] ima: Allocated hash algorithm: sha1 [ 3.663294] ima: No architecture policies found [ 3.665724] evm: Initialising EVM extended attributes: [ 3.667780] evm: security.selinux [ 3.668934] evm: security.ima [ 3.670022] evm: security.capability [ 3.671446] evm: HMAC attrs: 0x1 [ 3.674589] rtc_cmos 00:05: setting system clock to 2026-09-09 07:50:01 UTC (1788940201) [ 3.681252] debug: unmapping init [mem 0xffffffffac803000-0xffffffffac9fffff] [ 3.684649] debug: unmapping init [mem 0xffffffffab582000-0xffffffffab858fff] [ 3.693076] Write protecting the kernel read-only data: 28672k [ 3.696836] debug: unmapping init [mem 0xffffffffa9c03000-0xffffffffa9dfffff] [ 3.700287] debug: unmapping init [mem 0xffffffffaa514000-0xffffffffaa5fffff] [ 3.735895] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.745209] systemd[1]: Detected virtualization kvm. [ 3.746781] systemd[1]: Detected architecture x86-64. [ 3.748376] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.773415] systemd[1]: No hostname configured. [ 3.774723] systemd[1]: Set hostname to . [ 3.776336] random: systemd: uninitialized urandom read (16 bytes read) [ 3.778182] systemd[1]: Initializing machine ID from random generator. [ 3.835736] random: ln: uninitialized urandom read (6 bytes read) [ 3.928112] random: systemd: uninitialized urandom read (16 bytes read) [ 3.930922] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.934893] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.941582] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.912725] device-mapper: uevent: version 1.0.3 [ 4.916166] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.658190] virtio_net virtio0 ens2: renamed from eth0 [ 5.696364] random: fast init done [ 5.719960] scsi host0: ata_piix [ 5.728937] scsi host1: ata_piix [ 5.730576] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.733697] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.633323] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.177730] random: crng init done [ 10.179072] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.928392] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.243149] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.527693] SELinux: Disabled at runtime. [ 12.589281] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.598537] systemd[1]: Detected virtualization kvm. [ 12.600150] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.149709] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.155192] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.162180] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.166207] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.169423] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.179818] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.186448] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ 13.282742] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.910958] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.455215] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.502591] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.655531] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.676610] EDAC sbridge: Ver: 1.1.2 [ 16.446562] Key type dns_resolver registered [ 16.771840] NFS: Registering the id_resolver key type [ 16.774914] Key type id_resolver registered [ 16.777258] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg160-server login: [ 31.547143] hrtimer: interrupt took 2220323 ns [ 79.670051] libcfs: loading out-of-tree module taints kernel. [ 79.740382] Key type ._llcrypt registered [ 79.741736] Key type .llcrypt registered [ 79.908743] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 104.840943] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 107.431835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 107.479931] alg: No test for adler32 (adler32-zlib) [ 109.061223] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 110.174615] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 112.216183] Key type lgssc registered [ 114.723274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.224857] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.362216] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 214.658714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.689456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.067433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.100288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.219836] Lustre: lustre-MDT0000: new disk, initializing [ 216.325084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.355398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.689413] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.441698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 238.629901] Lustre: 6540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 238.671206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 238.681839] Lustre: Skipped 1 previous similar message [ 238.882797] Lustre: lustre-MDT0001: new disk, initializing [ 239.020644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.067498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.089273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 246.985879] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.523746] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 263.897651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.324425] Lustre: lustre-OST0000: new disk, initializing [ 264.336764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 264.361929] Lustre: 8476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.541525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.903989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.912364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 271.017584] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 271.035864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 287.246844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.529650] Lustre: lustre-OST0001: new disk, initializing [ 287.535239] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 287.541650] Lustre: 9550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.663804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 294.797357] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.616039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 297.636421] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.751498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 309.678923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.998665] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 331.725199] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing check_logdir /tmp/testlogs/ [ 337.822830] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing yml_node [ 343.850897] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 346.918901] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 350.391959] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 352.261691] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 03:55:47 EDT 2026 [ 375.917239] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 377.974725] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 397.793419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 397.808769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.821708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.353638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.354327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 400.382375] Lustre: Skipped 3 previous similar messages [ 401.782709] Lustre: server umount lustre-MDT0000 complete [ 405.741282] LustreError: 6530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940603 with bad export cookie 6862083777885043408 [ 405.749245] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 405.751068] LustreError: 6530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 410.593159] LustreError: 6548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.595162] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.608424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 410.618415] LustreError: 6548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 410.640454] Lustre: Skipped 3 previous similar messages [ 415.719403] LustreError: 8484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 415.719945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 415.735559] LustreError: 8484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 415.750818] Lustre: Skipped 2 previous similar messages [ 416.737641] LustreError: 6546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.749132] LustreError: 6546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 420.320836] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 420.631696] Lustre: server umount lustre-MDT0001 complete [ 424.844959] Lustre: server umount lustre-OST0000 complete [ 428.638627] Lustre: server umount lustre-OST0001 complete [ 436.464762] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 445.819972] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 483.030427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.275892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 483.302349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 483.455226] Lustre: lustre-MDT0000: new disk, initializing [ 483.542931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 483.553733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 489.290255] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 506.033060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.270546] Lustre: 15108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 506.480549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 506.496310] Lustre: Skipped 1 previous similar message [ 506.583449] Lustre: lustre-MDT0001: new disk, initializing [ 506.808578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 506.889365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 506.899225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 513.413568] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 525.312165] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 532.309807] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.562748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.894191] Lustre: lustre-OST0000: new disk, initializing [ 540.903881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 540.915173] Lustre: 17080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 541.073437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.573264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 542.590684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 542.706821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 549.408676] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 559.823646] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 565.777377] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 565.989490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 571.226944] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 571.447593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 578.530106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 578.530427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 578.538144] LustreError: Skipped 1 previous similar message [ 578.544676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 578.562405] Lustre: Skipped 1 previous similar message [ 578.573347] Lustre: Skipped 2 previous similar messages [ 579.872971] Lustre: server umount lustre-OST0000 complete [ 588.769792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.774936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.784897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.784906] Lustre: Skipped 1 previous similar message [ 590.467889] Lustre: server umount lustre-MDT0000 complete [ 593.889750] LustreError: 15121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.919865] LustreError: 15121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 595.226990] LustreError: 15103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940793 with bad export cookie 6862083777885044948 [ 595.239360] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.663528] Lustre: server umount lustre-MDT0001 complete [ 600.301548] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 03:59:55 (1788940795) [ 608.064761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 609.081052] LustreError: 19347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.222104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.354825] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 618.529384] LustreError: 19348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.563404] LustreError: 19348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 624.771527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.947872] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 637.728469] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 643.731614] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 652.147636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.851892] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 670.129563] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 679.398740] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 679.416077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.433876] Lustre: Skipped 1 previous similar message [ 679.439509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 679.462813] Lustre: Skipped 1 previous similar message [ 682.879441] Lustre: server umount lustre-OST0000 complete [ 694.380811] Lustre: server umount lustre-MDT0000 complete [ 695.779446] LustreError: 19343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 695.788112] LustreError: 19343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 698.837304] LustreError: 19330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940896 with bad export cookie 6862083777885045830 [ 698.860402] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.873702] LustreError: 19330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 699.332273] Lustre: server umount lustre-MDT0001 complete [ 711.276492] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 714.090498] Key type lgssc unregistered [ 714.276573] LNet: 22894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.281093] LNetError: 22894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.300712] LNet: Removed LNI 192.168.201.160@tcp [ 715.590276] Key type .llcrypt unregistered [ 715.594739] Key type ._llcrypt unregistered [ 743.060964] Key type ._llcrypt registered [ 743.062485] Key type .llcrypt registered [ 743.257316] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 744.243595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.330754] alg: No test for adler32 (adler32-zlib) [ 745.719104] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 745.940544] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 747.596054] Key type lgssc registered [ 749.209740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 759.000285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.015137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.689685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.128411] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 771.070266] LustreError: 24304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.086987] LustreError: 24304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 775.335184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.926836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 781.772585] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 791.868553] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 802.317346] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 812.182941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.568814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.359464] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 830.542702] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 831.073548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 851.436115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.439286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.450691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.457877] Lustre: Skipped 1 previous similar message [ 856.043442] Lustre: server umount lustre-OST0000 complete [ 861.630598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 861.632603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.657876] Lustre: Skipped 1 previous similar message [ 861.684460] Lustre: Skipped 2 previous similar messages [ 866.788604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.796776] Lustre: Skipped 1 previous similar message [ 868.054723] Lustre: server umount lustre-MDT0000 complete [ 871.917891] LustreError: 24304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.939352] LustreError: 24304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 872.544508] LustreError: 26150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941070 with bad export cookie 9267010315915420730 [ 872.551501] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.566493] LustreError: 26150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 872.990571] Lustre: server umount lustre-MDT0001 complete [ 886.419494] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 889.983507] Key type lgssc unregistered [ 890.276301] LNet: 28280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.286135] LNetError: 28280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.319144] LNet: Removed LNI 192.168.201.160@tcp [ 891.647706] Key type .llcrypt unregistered [ 891.652284] Key type ._llcrypt unregistered [ 924.204597] Key type ._llcrypt registered [ 924.210356] Key type .llcrypt registered [ 924.267692] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 925.208078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.271171] alg: No test for adler32 (adler32-zlib) [ 926.324880] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 926.570526] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 928.233209] Key type lgssc registered [ 929.829484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.108911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.129973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.558346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.685472] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 949.742532] LustreError: 29693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.762940] LustreError: 29693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 954.861030] LustreError: 29692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.678155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.999900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.987372] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 970.973722] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.656515] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 996.759087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 997.380242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.929262] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1009.677627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1022.203734] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1030.117736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1030.122371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.130943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.142183] Lustre: Skipped 1 previous similar message [ 1035.234379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1035.246221] Lustre: Skipped 2 previous similar messages [ 1036.244437] Lustre: server umount lustre-OST0000 complete [ 1043.943770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1043.958711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1043.997761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1045.476326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.871450] Lustre: server umount lustre-MDT0000 complete [ 1050.593748] LustreError: 31410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.605816] LustreError: 31410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1053.001689] LustreError: 29672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941250 with bad export cookie 2367201789342459945 [ 1053.009698] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.018634] LustreError: 29672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1053.382052] Lustre: server umount lustre-MDT0001 complete [ 1065.865206] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1068.810389] Key type lgssc unregistered [ 1069.218775] LNet: 33575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.224303] LNetError: 33575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.255255] LNet: Removed LNI 192.168.201.160@tcp [ 1070.229515] Key type .llcrypt unregistered [ 1070.231566] Key type ._llcrypt unregistered [ 1101.674678] Key type ._llcrypt registered [ 1101.676342] Key type .llcrypt registered [ 1101.745644] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1102.540131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1102.567226] alg: No test for adler32 (adler32-zlib) [ 1103.905890] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 1104.170760] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1105.840232] Key type lgssc registered [ 1106.991505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.359867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.400632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.850157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.000851] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1127.395249] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.414365] LustreError: 34984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1132.538406] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.877934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.549024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1145.419753] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1157.168666] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.102748] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1172.422581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1172.706262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.195548] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1187.148864] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1188.118741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1198.562737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.578296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.591459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1198.602590] Lustre: Skipped 1 previous similar message [ 1198.628529] Lustre: Skipped 1 previous similar message [ 1202.930308] Lustre: server umount lustre-OST0000 complete [ 1207.271572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1207.276321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.293530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.802959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1208.803110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.924529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1213.929329] Lustre: Skipped 2 previous similar messages [ 1219.049608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.053486] Lustre: Skipped 1 previous similar message [ 1221.088188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1221.249757] Lustre: server umount lustre-MDT0000 complete [ 1224.184077] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1224.228747] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1225.974561] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941423 with bad export cookie 15654044746747344861 [ 1225.988114] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1225.996381] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1226.738923] Lustre: server umount lustre-MDT0001 complete [ 1240.435777] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1243.973759] Key type lgssc unregistered [ 1244.306975] LNet: 38521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.315534] LNetError: 38521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.329962] LNet: Removed LNI 192.168.201.160@tcp [ 1245.447933] Key type .llcrypt unregistered [ 1245.450173] Key type ._llcrypt unregistered [ 1271.997091] Key type ._llcrypt registered [ 1272.003367] Key type .llcrypt registered [ 1272.227532] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1273.573468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1273.648034] alg: No test for adler32 (adler32-zlib) [ 1274.805983] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 1275.008825] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1276.640232] Key type lgssc registered [ 1277.694134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.886491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.895534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.324362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.564652] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1298.915428] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.937880] LustreError: 39931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1304.033870] LustreError: 39932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.357841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.750109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1310.242985] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1318.640305] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1327.383853] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1334.634592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1335.254314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1344.242305] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1351.873479] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1358.851331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1364.969473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1364.980221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.998691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1366.487604] Lustre: server umount lustre-OST0000 complete [ 1384.420695] Lustre: 39240:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941566/real 1788941566] req@00000000b43c7092 x1875841092974080/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788941582 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1384.470148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.538405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.559564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.657457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.687280] Lustre: Skipped 2 previous similar messages [ 1395.697556] Lustre: server umount lustre-MDT0000 complete [ 1399.793313] LustreError: 41971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.804561] LustreError: 41971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1400.926912] LustreError: 39914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941598 with bad export cookie 16544630597737783929 [ 1400.933040] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1400.945571] LustreError: 39914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1407.242923] Lustre: server umount lustre-MDT0001 complete [ 1420.779390] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1423.953849] Key type lgssc unregistered [ 1424.251638] LNet: 43641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.259241] LNetError: 43641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.277923] LNet: Removed LNI 192.168.201.160@tcp [ 1425.410521] Key type .llcrypt unregistered [ 1425.412382] Key type ._llcrypt unregistered [ 1452.897841] Key type ._llcrypt registered [ 1452.900239] Key type .llcrypt registered [ 1453.013670] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1454.101535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1454.325487] alg: No test for adler32 (adler32-zlib) [ 1455.492829] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 1455.857114] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1457.536277] Key type lgssc registered [ 1458.814642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1466.752732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1466.767819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.193517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.859232] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1477.605542] LustreError: 45051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1477.621751] LustreError: 45051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1482.416840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.724072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1482.939312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1487.903534] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1495.774060] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1504.631950] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1514.744538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1515.133121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.653360] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1532.161845] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1537.728462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1543.142923] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.150584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1543.158247] Lustre: Skipped 2 previous similar messages [ 1543.169142] Lustre: Skipped 2 previous similar messages [ 1545.383882] Lustre: server umount lustre-MDT0000 complete [ 1548.259447] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.302419] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1550.316763] LustreError: 45031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941748 with bad export cookie 6856982039206263126 [ 1550.337696] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1550.343648] LustreError: 45031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1553.378693] LustreError: 45047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.389514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.404587] LustreError: 45047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1553.429329] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1553.434980] Lustre: Skipped 1 previous similar message [ 1555.426713] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.446928] LustreError: 45046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1557.014782] Lustre: server umount lustre-MDT0001 complete [ 1578.529543] Lustre: server umount lustre-OST0000 complete [ 1595.589455] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1598.501859] Key type lgssc unregistered [ 1598.739641] LNet: 48910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.742898] LNetError: 48910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.783069] LNet: Removed LNI 192.168.201.160@tcp [ 1600.692608] Key type .llcrypt unregistered [ 1600.698565] Key type ._llcrypt unregistered [ 1627.818405] Key type ._llcrypt registered [ 1627.820929] Key type .llcrypt registered [ 1627.917578] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1629.643315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1629.706395] alg: No test for adler32 (adler32-zlib) [ 1630.622253] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 1630.787046] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1632.384643] Key type lgssc registered [ 1634.011218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.651286] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.688688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.384424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1665.504600] LustreError: 50295:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 1692.192204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1700.402125] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1708.573519] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1875.907464] Lustre: server umount lustre-OST0000 complete [ 1895.318628] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1898.019974] Key type lgssc unregistered [ 1898.399802] LNet: 51971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.412508] LNetError: 51971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.424608] LNet: Removed LNI 192.168.201.160@tcp [ 1899.210815] Key type .llcrypt unregistered [ 1899.216532] Key type ._llcrypt unregistered [ 1922.882561] Key type ._llcrypt registered [ 1922.884844] Key type .llcrypt registered [ 1922.982175] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1924.172973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.273343] alg: No test for adler32 (adler32-zlib) [ 1925.475137] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 1925.747552] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1927.387812] Key type lgssc registered [ 1928.846434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.541478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.587592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.326246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.603316] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1950.697720] LustreError: 53381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.718778] LustreError: 53381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1955.815255] LustreError: 53382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.291653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.069951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.308261] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1970.806069] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.094907] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.514578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.035412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.173335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1994.319068] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2004.344433] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2011.621689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.641514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.660444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.668204] Lustre: Skipped 1 previous similar message [ 2016.742331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.344950] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2021.484551] Lustre: server umount lustre-OST0000 complete [ 2026.978159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.984766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.985913] Lustre: Skipped 2 previous similar messages [ 2026.992702] Lustre: Skipped 1 previous similar message [ 2032.100336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2032.105298] Lustre: Skipped 2 previous similar messages [ 2037.225139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.236203] Lustre: Skipped 1 previous similar message [ 2039.264257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2039.733774] Lustre: server umount lustre-MDT0000 complete [ 2041.322308] LustreError: 54112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.378778] LustreError: 54112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2045.507545] LustreError: 53363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942243 with bad export cookie 7854816377806475455 [ 2045.543065] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.549041] LustreError: 53363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2046.141488] Lustre: server umount lustre-MDT0001 complete [ 2059.414963] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2062.397867] Key type lgssc unregistered [ 2062.616918] LNet: 56919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.628825] LNetError: 56919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.657463] LNet: Removed LNI 192.168.201.160@tcp [ 2064.894334] Key type .llcrypt unregistered [ 2064.900313] Key type ._llcrypt unregistered [ 2090.597651] Key type ._llcrypt registered [ 2090.599891] Key type .llcrypt registered [ 2090.708277] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2091.775126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2092.002119] alg: No test for adler32 (adler32-zlib) [ 2093.055632] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2093.223866] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2094.864226] Key type lgssc registered [ 2096.170360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2104.222859] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2104.235535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2119.777891] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2124.898445] LustreError: 58305:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 2151.520828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.055240] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2165.754437] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2175.621036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.277314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.376521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2181.602364] LustreError: 58330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.618953] LustreError: 58330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2182.418319] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2186.730084] LustreError: 58331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.845916] LustreError: 59340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.876880] LustreError: 59340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2192.202749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.707433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.951847] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2208.124262] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.380865] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2217.442294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.447341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.456909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.468469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2218.471945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2218.475241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2218.491169] Lustre: Skipped 1 previous similar message [ 2222.857518] Lustre: server umount lustre-OST0000 complete [ 2249.189972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.192779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2249.237833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.273284] Lustre: Skipped 1 previous similar message [ 2254.311720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2254.324751] Lustre: Skipped 1 previous similar message [ 2254.961388] Lustre: server umount lustre-MDT0000 complete [ 2258.413413] LustreError: 59347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.453100] LustreError: 59347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2259.157330] LustreError: 58311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942456 with bad export cookie 9506688129005503337 [ 2259.161870] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.172621] LustreError: 58311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2265.740978] Lustre: server umount lustre-MDT0001 complete [ 2280.330837] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2283.541427] Key type lgssc unregistered [ 2283.826565] LNet: 62054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2283.834657] LNetError: 62054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.853063] LNet: Removed LNI 192.168.201.160@tcp [ 2284.963940] Key type .llcrypt unregistered [ 2284.968120] Key type ._llcrypt unregistered [ 2309.028176] Key type ._llcrypt registered [ 2309.029612] Key type .llcrypt registered [ 2309.103873] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2310.700366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.728089] alg: No test for adler32 (adler32-zlib) [ 2311.946842] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2312.384436] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2314.152378] Key type lgssc registered [ 2315.632839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2325.746152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2325.756863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2327.238890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.245436] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2337.762306] LustreError: 63465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.780409] LustreError: 63465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2341.343650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.685247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2346.440666] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2354.640396] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.686914] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.895897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.200148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.340164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2375.201784] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2383.751327] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.278656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.279972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2402.311123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2407.402705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2407.409579] Lustre: Skipped 1 previous similar message [ 2408.466104] Lustre: server umount lustre-OST0000 complete [ 2413.537291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2413.551379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.580196] Lustre: Skipped 1 previous similar message [ 2413.586577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2413.589263] Lustre: Skipped 1 previous similar message [ 2417.635676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2417.636276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.486438] Lustre: server umount lustre-MDT0000 complete [ 2422.764844] LustreError: 63466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2422.778918] LustreError: 63466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2424.134427] LustreError: 65309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942621 with bad export cookie 3961708855959012990 [ 2424.135567] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2424.149864] LustreError: 65309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2424.508580] Lustre: server umount lustre-MDT0001 complete [ 2436.835460] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2439.763570] Key type lgssc unregistered [ 2440.050127] LNet: 67019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.061111] LNetError: 67019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.074113] LNet: Removed LNI 192.168.201.160@tcp [ 2441.162590] Key type .llcrypt unregistered [ 2441.165643] Key type ._llcrypt unregistered [ 2468.323478] Key type ._llcrypt registered [ 2468.327962] Key type .llcrypt registered [ 2468.485664] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2469.690823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2469.854475] alg: No test for adler32 (adler32-zlib) [ 2470.902258] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2471.155773] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2472.864271] Key type lgssc registered [ 2474.055924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2483.685218] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:31:19 (1788942679) [ 2490.748032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2490.767239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.140081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.164485] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2501.616712] LustreError: 68717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.647957] LustreError: 68717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2506.495624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.724449] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2506.917352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2511.818969] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2520.989643] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.341046] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2534.780606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.206583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.247412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2541.839662] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2551.193441] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.739795] Lustre: Failing over lustre-MDT0000 [ 2556.899593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.906351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2556.918476] Lustre: Skipped 1 previous similar message [ 2556.937572] Lustre: Skipped 2 previous similar messages [ 2557.133175] Lustre: server umount lustre-MDT0000 complete [ 2558.110974] LustreError: 70006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.021980] LustreError: 68714:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.057037] LustreError: 68714:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2567.055163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.141844] LustreError: 68713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.168388] LustreError: 68713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2567.263988] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2567.494617] Lustre: *** cfs_fail_loc=135, val=0*** [ 2567.645594] LustreError: 71807:0:(obd_config.c:920:class_setup()) setup lustre-MDT0000 failed (-2) [ 2567.656748] LustreError: 71807:0:(obd_config.c:2201:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -2 [ 2567.673923] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2567.673923] [ 2567.684060] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2567.700080] LustreError: 71795:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2567.715421] LustreError: 71795:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2567.727743] LustreError: 71795:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2567.743094] LustreError: 71795:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2567.787034] Lustre: server umount lustre-MDT0000 complete [ 2567.792110] LustreError: 71795:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2572.296158] LustreError: 68717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.317896] LustreError: 68717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2574.841666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.016356] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2575.024945] LustreError: 68710:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2575.029460] LustreError: 72152:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.160@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000004a3fd381) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2575.412859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.501723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2579.569030] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2580.467789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2580.495474] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2580.500256] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2580.555388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2590.699822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.706381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.726594] Lustre: Skipped 2 previous similar messages [ 2590.757348] Lustre: Skipped 1 previous similar message [ 2594.886668] Lustre: server umount lustre-OST0000 complete [ 2600.931430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.942708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.948736] Lustre: Skipped 1 previous similar message [ 2604.871781] Lustre: server umount lustre-MDT0000 complete [ 2606.051986] LustreError: 68713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.069526] LustreError: 68713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2608.532544] LustreError: 68698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942806 with bad export cookie 2827447560817746525 [ 2608.535112] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.536538] LustreError: 68698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2608.753650] Lustre: server umount lustre-MDT0001 complete [ 2621.897033] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2625.213917] Key type lgssc unregistered [ 2625.461650] LNet: 73588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2625.470315] LNetError: 73588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2625.487164] LNet: Removed LNI 192.168.201.160@tcp [ 2626.397475] Key type .llcrypt unregistered [ 2626.399034] Key type ._llcrypt unregistered [ 2651.659245] Key type ._llcrypt registered [ 2651.663724] Key type .llcrypt registered [ 2651.810876] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2652.878652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.113709] alg: No test for adler32 (adler32-zlib) [ 2654.195357] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2654.386720] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2656.080276] Key type lgssc registered [ 2657.645546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.767576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.781684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.456710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2672.661810] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2677.736149] LustreError: 74997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.751983] LustreError: 74997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2681.913991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.382226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.396762] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2695.446364] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2701.617950] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2709.724471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.168757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.259984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2717.089528] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2726.353353] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2733.312883] Lustre: Failing over lustre-MDT0000 [ 2733.537817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2733.546281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2733.565899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2733.696250] LustreError: 74994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.932505] Lustre: server umount lustre-MDT0000 complete [ 2738.160861] LustreError: 74998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.188461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.220825] LustreError: 74998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2738.262893] Lustre: Skipped 1 previous similar message [ 2743.277930] LustreError: 74997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.289720] LustreError: 74997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2744.896770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.128544] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.369419] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2745.507738] LustreError: 78086:0:(obd_config.c:920:class_setup()) setup lustre-MDT0000 failed (-9) [ 2745.527254] LustreError: 78086:0:(obd_config.c:2201:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -9 [ 2745.538814] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2745.538814] [ 2745.551249] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2745.565652] LustreError: 78075:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2745.573862] LustreError: 78075:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2745.580974] LustreError: 78075:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2745.588438] LustreError: 78075:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2745.605878] Lustre: server umount lustre-MDT0000 complete [ 2745.608410] LustreError: 78075:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2748.388347] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.401276] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2753.152874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.303665] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.313289] LustreError: 74989:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2753.578877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.649752] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2754.154193] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2758.647411] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2758.691721] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2758.705985] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2758.739817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2768.883450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2768.890532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2768.905614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2768.917501] Lustre: Skipped 1 previous similar message [ 2773.770223] Lustre: server umount lustre-OST0000 complete [ 2779.105179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2779.109507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.112093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.124859] Lustre: Skipped 2 previous similar messages [ 2779.139726] Lustre: Skipped 1 previous similar message [ 2783.923401] Lustre: server umount lustre-MDT0000 complete [ 2784.230300] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.261444] LustreError: 75725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2787.836568] LustreError: 77579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942985 with bad export cookie 13146674244043143482 [ 2787.847695] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.850230] LustreError: 77579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2788.169636] Lustre: server umount lustre-MDT0001 complete [ 2801.452119] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2804.202681] Key type lgssc unregistered [ 2804.469786] LNet: 79865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2804.480371] LNetError: 79865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.543398] LNet: Removed LNI 192.168.201.160@tcp [ 2806.758555] Key type .llcrypt unregistered [ 2806.761269] Key type ._llcrypt unregistered [ 2822.597439] Key type ._llcrypt registered [ 2822.601347] Key type .llcrypt registered [ 2822.845337] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2823.420554] Key type .llcrypt unregistered [ 2823.422776] Key type ._llcrypt unregistered [ 2852.212238] Key type ._llcrypt registered [ 2852.217379] Key type .llcrypt registered [ 2852.306957] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2853.613706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2853.662551] alg: No test for adler32 (adler32-zlib) [ 2854.679196] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2854.853576] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2856.504919] Key type lgssc registered [ 2857.630371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2867.222476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2867.231227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.630233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.694597] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2878.949821] LustreError: 81761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.973725] LustreError: 81761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2883.048592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.496391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2889.534301] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2897.891996] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2903.915439] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2911.504947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2911.944571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.989759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2919.694732] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2928.253708] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2938.853716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2938.859321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2938.861824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.889112] Lustre: Skipped 1 previous similar message [ 2943.756695] Lustre: server umount lustre-OST0000 complete [ 2949.095484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2949.092600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.108455] Lustre: Skipped 1 previous similar message [ 2949.150600] Lustre: Skipped 1 previous similar message [ 2954.210343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2954.218232] Lustre: Skipped 2 previous similar messages [ 2958.317645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.400145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.696138] Lustre: server umount lustre-MDT0000 complete [ 2963.432056] LustreError: 82490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.456842] LustreError: 82490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2967.081833] LustreError: 81741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943164 with bad export cookie 9584843264599131700 [ 2967.097500] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.101215] LustreError: 81741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2967.723975] Lustre: server umount lustre-MDT0001 complete [ 2979.821420] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2983.332660] Key type lgssc unregistered [ 2983.651661] LNet: 85296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2983.659463] LNetError: 85296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2983.680791] LNet: Removed LNI 192.168.201.160@tcp [ 2984.766713] Key type .llcrypt unregistered [ 2984.769127] Key type ._llcrypt unregistered [ 3010.818403] Key type ._llcrypt registered [ 3010.820809] Key type .llcrypt registered [ 3011.003520] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3012.742734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3012.804766] alg: No test for adler32 (adler32-zlib) [ 3014.080693] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3014.571630] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3016.376203] Key type lgssc registered [ 3017.780516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3024.943874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3024.982301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.485734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3033.005828] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3036.650622] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.659541] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3041.776800] LustreError: 86711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.997110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.895836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.968988] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3059.814604] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3066.988797] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3074.537303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.840719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.387757] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3088.779185] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3096.380930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 3106.806672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.813428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.818056] Lustre: Skipped 1 previous similar message [ 3106.821370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.907838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3111.912151] Lustre: Skipped 1 previous similar message [ 3112.755644] Lustre: server umount lustre-OST0000 complete [ 3121.634511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3121.646728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.671539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3121.681473] Lustre: Skipped 1 previous similar message [ 3123.595512] Lustre: server umount lustre-MDT0000 complete [ 3127.267762] LustreError: 86710:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.136118] LustreError: 86692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943325 with bad export cookie 14745768863948525442 [ 3128.138796] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.143851] LustreError: 86692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3128.633578] Lustre: server umount lustre-MDT0001 complete [ 3140.814362] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3144.514727] Key type lgssc unregistered [ 3144.832519] LNet: 90261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3144.836960] LNetError: 90261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3144.851599] LNet: Removed LNI 192.168.201.160@tcp [ 3145.965116] Key type .llcrypt unregistered [ 3145.966852] Key type ._llcrypt unregistered [ 3174.799989] Key type ._llcrypt registered [ 3174.807950] Key type .llcrypt registered [ 3174.924880] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3176.139372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.163084] alg: No test for adler32 (adler32-zlib) [ 3177.248676] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3177.571713] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3179.260996] Key type lgssc registered [ 3180.647884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3190.670932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3190.690927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.348601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3197.728438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3202.552802] LustreError: 91674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.571825] LustreError: 91674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3207.609487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.651363] LustreError: 91675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3207.969679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3213.401531] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3221.267232] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3228.829647] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.464286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.086397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.147783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3243.190396] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3251.057861] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.970469] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3263.975055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3263.995375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.067302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3268.071643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.083723] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.232830] Lustre: server umount lustre-OST0000 complete [ 3278.310536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.322622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3278.326039] Lustre: Skipped 1 previous similar message [ 3278.337963] Lustre: Skipped 2 previous similar messages [ 3283.426175] LustreError: 92642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.578171] Lustre: server umount lustre-MDT0000 complete [ 3288.236125] LustreError: 94167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943486 with bad export cookie 3973306519096385182 [ 3288.238431] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.254444] LustreError: 94167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3288.819940] Lustre: server umount lustre-MDT0001 complete [ 3303.285450] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3307.075773] Key type lgssc unregistered [ 3307.471686] LNet: 95228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3307.479612] LNetError: 95228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3307.495235] LNet: Removed LNI 192.168.201.160@tcp [ 3308.811132] Key type .llcrypt unregistered [ 3308.819993] Key type ._llcrypt unregistered [ 3341.212357] Key type ._llcrypt registered [ 3341.214322] Key type .llcrypt registered [ 3341.367442] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3343.092212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.111767] alg: No test for adler32 (adler32-zlib) [ 3344.189866] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3344.408522] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3346.104175] Key type lgssc registered [ 3347.411723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3357.227127] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3357.256986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.833897] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3377.952309] LustreError: 96615:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 3404.704211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.758176] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3421.835454] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3430.385619] Lustre: server umount lustre-OST0000 complete [ 3442.005531] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:47:17 (1788943637) [ 3450.541524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.173235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3456.480617] LustreError: 98256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3456.503008] LustreError: 98256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3457.439854] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3460.581756] LustreError: 98257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.698582] LustreError: 98256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.866614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.664524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3476.059787] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3486.071287] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3493.623422] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3502.634072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3503.136691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.711875] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3520.586894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3521.756804] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3540.450478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3540.461997] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3540.472231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3540.966803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3541.006876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.381741] Lustre: server umount lustre-OST0000 complete [ 3551.204285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.210404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.235426] Lustre: Skipped 1 previous similar message [ 3551.262423] Lustre: Skipped 1 previous similar message [ 3555.055553] Lustre: server umount lustre-MDT0000 complete [ 3556.338590] LustreError: 98256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.373561] LustreError: 98256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3560.065729] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943757 with bad export cookie 13018761145683816924 [ 3560.071217] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.085116] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3560.385422] Lustre: server umount lustre-MDT0001 complete [ 3573.885960] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3577.412807] Key type lgssc unregistered [ 3577.860239] LNet: 101950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.872027] LNetError: 101950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.904072] LNet: Removed LNI 192.168.201.160@tcp [ 3579.028190] Key type .llcrypt unregistered [ 3579.030304] Key type ._llcrypt unregistered [ 3629.094263] Key type ._llcrypt registered [ 3629.096530] Key type .llcrypt registered [ 3629.301954] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3631.025675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3631.125069] alg: No test for adler32 (adler32-zlib) [ 3632.229062] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3632.470606] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3634.151966] Key type lgssc registered [ 3635.328920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3643.952432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3643.984190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.524911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.845149] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3655.649559] LustreError: 104134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.674926] LustreError: 104134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3660.560677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3660.773551] LustreError: 104135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.053646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3667.171723] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3677.831639] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3685.732913] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3694.418448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.747238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3702.066895] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3712.082604] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3715.311831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3730.920806] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.922759] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3730.951766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3732.417295] Lustre: server umount lustre-OST0000 complete [ 3738.085307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3738.091329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.102596] Lustre: Skipped 1 previous similar message [ 3738.108464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3738.110421] Lustre: Skipped 1 previous similar message [ 3741.154321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3741.154451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.450340] Lustre: server umount lustre-MDT0000 complete [ 3746.277790] LustreError: 104129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3746.311318] LustreError: 104129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3747.925830] LustreError: 104117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943945 with bad export cookie 6287703207189280751 [ 3747.940586] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.648422] Lustre: server umount lustre-MDT0001 complete [ 3761.464816] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3764.184908] Key type lgssc unregistered [ 3764.445904] LNet: 107698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3764.457552] LNetError: 107698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3764.479742] LNet: Removed LNI 192.168.201.160@tcp [ 3765.764471] Key type .llcrypt unregistered [ 3765.767433] Key type ._llcrypt unregistered [ 3788.232876] Key type ._llcrypt registered [ 3788.238200] Key type .llcrypt registered [ 3788.394626] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3789.714599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.762079] alg: No test for adler32 (adler32-zlib) [ 3791.247320] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3791.551087] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3793.232682] Key type lgssc registered [ 3794.451022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3801.770747] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3801.817661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3817.376729] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3822.497557] LustreError: 108936:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 3849.248238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3856.830726] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3865.292431] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3873.573652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.179927] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3874.205431] LustreError: 109959:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3874.233457] LustreError: 109959:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3874.245870] LustreError: 109959:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3874.515922] Lustre: server umount lustre-MDT0000 complete [ 3874.525656] LustreError: 109959:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3892.192208] Lustre: 108273:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788944074/real 1788944074] req@000000008aaf6180 x1875843732213120/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788944090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3892.214570] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.224404] LustreError: 108948:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3912.868647] Lustre: server umount lustre-OST0000 complete [ 3925.558815] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 3939.115794] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3982.649690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3982.954866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3982.985990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3983.068324] Lustre: lustre-MDT0000: new disk, initializing [ 3983.157383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.176733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3989.786418] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4002.505873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.648230] Lustre: 112757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.729441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4002.737347] Lustre: Skipped 1 previous similar message [ 4002.803651] Lustre: lustre-MDT0001: new disk, initializing [ 4002.885541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.929343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4002.942203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4008.773200] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4021.309630] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4027.724813] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4037.104457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4037.437974] Lustre: lustre-OST0000: new disk, initializing [ 4037.443085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4037.455772] Lustre: 114726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4037.594127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.877949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4044.900988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4045.068033] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4045.928816] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4058.552667] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4064.962554] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4065.283084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4071.110115] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4071.319516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4076.011410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4076.022813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.030601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4076.034517] LustreError: Skipped 1 previous similar message [ 4076.094432] Lustre: Skipped 1 previous similar message [ 4079.020439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4079.031905] Lustre: Skipped 1 previous similar message [ 4079.937754] Lustre: server umount lustre-OST0000 complete [ 4089.314899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.346482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.356355] Lustre: Skipped 2 previous similar messages [ 4091.539298] Lustre: server umount lustre-MDT0000 complete [ 4096.904306] LustreError: 113550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944294 with bad export cookie 17933956740704262673 [ 4096.906267] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4096.924745] LustreError: 113550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4097.381250] Lustre: server umount lustre-MDT0001 complete [ 4109.829599] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 04:58:25 (1788944305) [ 4114.882635] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4134.226750] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 4143.967138] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 4181.685282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.977415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4182.003817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4182.072651] Lustre: lustre-MDT0000: new disk, initializing [ 4182.111704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.123634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4187.494684] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4201.568886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.752715] Lustre: 119631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4201.801603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4201.803945] Lustre: Skipped 1 previous similar message [ 4201.857777] Lustre: lustre-MDT0001: new disk, initializing [ 4201.932230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4201.972094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4201.985339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4207.971081] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4219.751381] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4227.492861] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4235.937140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4236.354700] Lustre: lustre-OST0000: new disk, initializing [ 4236.362825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4236.391175] Lustre: 121600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4236.528404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.043492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4243.055904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4243.224918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4244.133509] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4253.898623] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4259.991625] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4260.430952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4266.179825] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4266.486207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.156080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.166097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.182523] Lustre: Skipped 2 previous similar messages [ 4274.190242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.201171] Lustre: Skipped 1 previous similar message [ 4275.440340] Lustre: server umount lustre-OST0000 complete [ 4283.880796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4283.893301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.914840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.593656] Lustre: server umount lustre-MDT0000 complete [ 4289.506350] LustreError: 119641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.529248] LustreError: 119641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4290.994118] LustreError: 119622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944488 with bad export cookie 17933956740704263555 [ 4290.996102] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.017203] LustreError: 119622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4291.561684] Lustre: server umount lustre-MDT0001 complete [ 4301.709029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.335932] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.462077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.949191] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.634309] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4313.068983] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.187187] LustreError: 123831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4319.480697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.613489] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4333.303806] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4339.797317] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4348.325513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4356.494977] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4366.361274] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4378.181992] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4382.694768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.707833] Lustre: Skipped 1 previous similar message [ 4382.708026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4382.713816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.713828] Lustre: Skipped 2 previous similar messages [ 4388.664628] Lustre: server umount lustre-OST0000 complete [ 4396.515109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4396.528273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.545827] Lustre: Skipped 1 previous similar message [ 4399.725334] Lustre: server umount lustre-MDT0000 complete [ 4403.172561] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.210680] LustreError: 123830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4404.086119] LustreError: 123813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944601 with bad export cookie 17933956740704264437 [ 4404.096778] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.101103] LustreError: 123813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4404.599452] Lustre: server umount lustre-MDT0001 complete [ 4416.286768] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 4419.360628] Key type lgssc unregistered [ 4419.633622] LNet: 127460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4419.645845] LNetError: 127460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4419.665792] LNet: Removed LNI 192.168.201.160@tcp [ 4420.788956] Key type .llcrypt unregistered [ 4420.793518] Key type ._llcrypt unregistered [ 4441.600239] Key type ._llcrypt registered [ 4441.607831] Key type .llcrypt registered [ 4441.709884] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 4458.921967] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 4460.200642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4460.246237] alg: No test for adler32 (adler32-zlib) [ 4461.381814] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 4461.861386] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 4463.632184] Key type lgssc registered [ 4464.795551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4505.161609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4505.202955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.541339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4506.641758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4506.836724] Lustre: lustre-MDT0000: new disk, initializing [ 4506.951111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4506.972841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4512.317387] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4524.452638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.587802] Lustre: 130610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4524.704508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4524.711224] Lustre: Skipped 1 previous similar message [ 4524.890451] Lustre: lustre-MDT0001: new disk, initializing [ 4525.140380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4525.201363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4525.214782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4530.597673] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4542.675797] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4549.710535] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4557.388897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4557.698665] Lustre: lustre-OST0000: new disk, initializing [ 4557.701982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4557.706816] Lustre: 132579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4557.819750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.046961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4564.060557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4564.203769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4565.427528] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4577.142336] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4583.170893] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4583.463573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4588.147424] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4588.342248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4595.169150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4595.170221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.186204] LustreError: Skipped 1 previous similar message [ 4595.193215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4595.199441] Lustre: Skipped 1 previous similar message [ 4595.241436] Lustre: Skipped 1 previous similar message [ 4597.130366] Lustre: server umount lustre-OST0000 complete [ 4601.826759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4601.835714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.857258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4605.411135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.415630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4605.453695] Lustre: Skipped 1 previous similar message [ 4608.057895] Lustre: server umount lustre-MDT0000 complete [ 4612.352587] LustreError: 130603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944810 with bad export cookie 14923348307922106147 [ 4612.354293] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.361461] LustreError: 130603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4613.150935] Lustre: server umount lustre-MDT0001 complete [ 4622.467432] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:06:58 (1788944818) [ 4630.076966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.572987] LustreError: 134993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.590191] LustreError: 134993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4630.643302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.135247] LustreError: 134994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.357638] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4640.235683] LustreError: 134993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4645.350436] LustreError: 134994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.470886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.799514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4651.601838] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4659.398025] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4665.208613] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4667.361622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.368446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4667.392582] Lustre: Skipped 1 previous similar message [ 4667.419478] Lustre: Skipped 1 previous similar message [ 4671.522557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4671.540943] Lustre: Skipped 1 previous similar message [ 4673.755892] Lustre: server umount lustre-MDT0000 complete [ 4676.589751] LustreError: 134993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.616953] LustreError: 134993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4678.270853] LustreError: 134974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944876 with bad export cookie 14923348307922107036 [ 4678.307526] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.704352] Lustre: server umount lustre-MDT0001 complete [ 4688.513162] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:08:04 (1788944884) [ 4695.443736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.136815] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4716.257093] LustreError: 137413:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 4741.984576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.233521] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4757.756443] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4760.585978] Lustre: server umount lustre-OST0000 complete [ 4769.580797] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:09:25 (1788944965) [ 4776.540802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.101517] LustreError: 138859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.185848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.117651] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4792.152815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4797.832537] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4806.136154] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.403921] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4819.196684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.552249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.557943] Lustre: Skipped 1 previous similar message [ 4825.805849] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4837.810586] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4848.614714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.616233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4848.642695] Lustre: Skipped 1 previous similar message [ 4848.665932] Lustre: Skipped 2 previous similar messages [ 4852.863322] Lustre: server umount lustre-MDT0000 complete [ 4853.736216] LustreError: 138860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.772244] LustreError: 138860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4856.302546] LustreError: 138840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945054 with bad export cookie 14923348307922107701 [ 4856.308207] LustreError: 138840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4856.314115] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4856.568219] Lustre: server umount lustre-MDT0001 complete [ 4872.003767] Lustre: server umount lustre-OST0000 complete [ 4882.634708] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:11:18 (1788945078) [ 4890.184334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.733621] LustreError: 142515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.793965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.001302] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4904.490197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.199169] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4918.723918] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4924.927689] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4933.222569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4935.630740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4940.652906] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4949.002617] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4955.323811] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4955.607282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4960.591684] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4960.842202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4965.347080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.349914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4965.376068] Lustre: Skipped 2 previous similar messages [ 4965.398818] Lustre: Skipped 1 previous similar message [ 4969.229711] Lustre: server umount lustre-OST0000 complete [ 4987.361968] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4987.572794] Lustre: server umount lustre-MDT0000 complete [ 4990.945668] LustreError: 142515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.970260] LustreError: 142515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4992.231265] LustreError: 142495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945190 with bad export cookie 14923348307922108772 [ 4992.244270] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.249953] LustreError: 142495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4992.793476] Lustre: server umount lustre-MDT0001 complete [ 5003.393741] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:13:18 (1788945198) [ 5011.227332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5026.785660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5031.905361] LustreError: 146676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 5057.633816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.636849] Lustre: Skipped 2 previous similar messages [ 5065.792608] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5074.186682] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5082.215327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.770442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5087.498946] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5096.042803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5101.803446] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5110.329338] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5117.240682] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5122.348243] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5122.618367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5127.794736] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5127.960296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5132.261364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5132.264787] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.266281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5132.266288] Lustre: Skipped 6 previous similar messages [ 5132.288453] Lustre: Skipped 4 previous similar messages [ 5136.385991] Lustre: server umount lustre-MDT0000 complete [ 5137.387318] LustreError: 146701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.421049] LustreError: 146701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5140.874572] LustreError: 148458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945338 with bad export cookie 14923348307922109794 [ 5140.876884] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.894570] LustreError: 148458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5141.385158] Lustre: server umount lustre-MDT0001 complete [ 5156.335908] Lustre: server umount lustre-OST0000 complete [ 5165.767656] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:16:01 (1788945361) [ 5174.018457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5189.730875] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5194.849368] LustreError: 150874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 5227.740973] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5236.150929] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5243.701664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.890800] LustreError: 128831:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000eef07fcd x1875844434918784/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5244.311151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5249.136772] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5257.085443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.309615] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5269.365870] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5278.587679] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5287.192642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5287.392825] Lustre: lustre-OST0001: new disk, initializing [ 5287.397701] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5287.404427] Lustre: 153748:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5294.677752] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5296.249226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5296.270219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5296.408751] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5306.516258] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5312.456290] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5312.781966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5317.563720] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5317.780700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5321.192813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5321.207848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.247548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5321.249930] Lustre: Skipped 2 previous similar messages [ 5326.050196] Lustre: server umount lustre-OST0000 complete [ 5332.456413] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5344.224152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5344.585565] Lustre: server umount lustre-OST0001 complete [ 5349.877597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5349.889807] LustreError: Skipped 1 previous similar message [ 5355.617600] Lustre: server umount lustre-MDT0000 complete [ 5359.763884] LustreError: 150880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945557 with bad export cookie 14923348307922110746 [ 5359.766037] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5359.778269] LustreError: 150880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5360.216733] Lustre: server umount lustre-MDT0001 complete [ 5385.036951] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:19:40 (1788945580) [ 5386.865103] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5388.543792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5390.397816] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:19:45 (1788945585) [ 5398.455858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.895468] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5398.972944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5399.301143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.310504] Lustre: Skipped 6 previous similar messages [ 5404.613315] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5415.042764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.143249] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5415.203688] Lustre: 157313:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5421.751882] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5431.697222] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5440.994877] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5448.776857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5448.912992] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5455.608050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5456.382863] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5464.600594] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5469.676888] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5469.954087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5475.147391] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5475.395985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5481.467199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5481.472136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.479745] LustreError: Skipped 1 previous similar message [ 5481.482367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5481.515670] Lustre: Skipped 6 previous similar messages [ 5481.536705] Lustre: Skipped 12 previous similar messages [ 5483.857068] Lustre: server umount lustre-OST0000 complete [ 5511.894370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5518.892149] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5521.508908] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.514265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.519755] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5521.533985] Lustre: Skipped 1 previous similar message [ 5529.455258] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5536.937846] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5537.389437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5543.298937] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5543.580620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5551.041690] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8dc1498e1000.ost_server_uuid 50 [ 5553.084434] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8dc1498e1000.ost_server_uuid in FULL state after 0 sec [ 5553.385588] LustreError: 158871:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.60@tcp) returned error from blocking AST (req@00000000cc2da4d7 x1875844435037952 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000d63cee60/0xcf1a6238011fd274 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.60@tcp remote: 0x2fc8f29db3b34757 expref: 5 pid: 158871 timeout: 5653 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5553.448393] LustreError: lustre-MDT0000: A client on nid 192.168.201.60@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5553.465483] LustreError: 157309:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.60@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000d63cee60/0xcf1a6238011fd274 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.60@tcp remote: 0x2fc8f29db3b34757 expref: 6 pid: 158871 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5558.760747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5565.089176] Lustre: server umount lustre-OST0000 complete [ 5575.671405] Lustre: server umount lustre-MDT0000 complete [ 5578.213033] LustreError: 157324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.227791] LustreError: 157324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 5580.143915] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945777 with bad export cookie 14923348307922111922 [ 5580.169887] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.185838] LustreError: 157304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5586.693219] Lustre: server umount lustre-MDT0001 complete [ 5599.677433] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5602.657390] Key type lgssc unregistered [ 5602.898844] LNet: 163432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5602.921275] LNetError: 163432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5602.948309] LNet: Removed LNI 192.168.201.160@tcp [ 5604.259258] Key type .llcrypt unregistered [ 5604.261169] Key type ._llcrypt unregistered [ 5630.484183] Key type ._llcrypt registered [ 5630.485680] Key type .llcrypt registered [ 5630.588067] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5631.498582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5631.787729] alg: No test for adler32 (adler32-zlib) [ 5632.792994] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 5632.989321] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5634.601299] Key type lgssc registered [ 5636.406394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5648.343952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5648.396701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.114090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5656.109970] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5659.620881] LustreError: 164846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5659.640795] LustreError: 164846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5664.738187] LustreError: 164847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.239606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5665.527196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.434792] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5678.571506] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5684.687843] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5691.311396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.664645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.811695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5698.376834] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5707.482716] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5717.243793] Lustre: Failing over lustre-MDT0000 [ 5717.649596] Lustre: server umount lustre-MDT0000 complete [ 5721.575183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.588765] LustreError: 164846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.591499] Lustre: Skipped 2 previous similar messages [ 5721.620139] LustreError: 164846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5726.693670] LustreError: 165574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5726.710787] LustreError: 165574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5730.224111] LustreError: 164828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.60@tcp arrived at 1788945928 with bad export cookie 7509379793863119371 [ 5730.289866] LustreError: 164828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5731.813762] LustreError: 164847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5731.847143] LustreError: 164847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5737.952568] Lustre: 164155:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788945919/real 1788945919] req@000000004bfa49bc x1875845662666880/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788945935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5737.979308] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.673710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.075113] LustreError: 165574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.117376] LustreError: 165574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5747.234118] LustreError: 164152:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb6dac80 x1875845662672384/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5747.675705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5747.739180] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5752.822203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5752.825774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5754.213437] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5756.513499] Lustre: lustre-MDT0000: Denying connection for new client 74c5d2a5-a3ca-4dce-9c3f-09e5c122e9b4 (at 192.168.201.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 5767.651384] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5767.660218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.678433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.402287] Lustre: server umount lustre-OST0000 complete [ 5775.410431] LustreError: 168578:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5775.422977] Lustre: 167927:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5775.439944] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5775.457746] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5775.465607] LustreError: 167927:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008457a6a3 x1875845662687488/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5775.478341] LustreError: 167927:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5775.488426] LustreError: 167927:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5775.504891] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5775.787556] Lustre: server umount lustre-MDT0000 complete [ 5778.409348] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5778.425817] Lustre: Skipped 1 previous similar message [ 5778.435636] LustreError: 167540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.458936] LustreError: 167540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5779.619186] LustreError: 164827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945977 with bad export cookie 7509379793863119476 [ 5779.625944] LustreError: 164152:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5779.629453] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.055767] Lustre: server umount lustre-MDT0001 complete [ 5793.884671] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5796.846752] Key type lgssc unregistered [ 5797.138617] LNet: 169329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5797.144678] LNetError: 169329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5797.151677] LNet: Removed LNI 192.168.201.160@tcp [ 5797.993213] Key type .llcrypt unregistered [ 5797.995770] Key type ._llcrypt unregistered [ 5824.849196] Key type ._llcrypt registered [ 5824.852254] Key type .llcrypt registered [ 5824.940862] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5826.227424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5826.439764] alg: No test for adler32 (adler32-zlib) [ 5827.530544] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 5827.814756] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5829.568186] Key type lgssc registered [ 5831.177160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5840.302084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5840.332043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5841.840320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5846.757308] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5852.140323] LustreError: 170740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.161605] LustreError: 170740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5855.885079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.229345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5861.503121] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5870.469785] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5878.748748] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5885.066987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5885.511952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5886.594160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5892.333452] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5901.937652] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5911.523681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5911.542057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5911.562738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5911.564902] Lustre: Skipped 1 previous similar message [ 5911.581704] Lustre: Skipped 1 previous similar message [ 5916.643045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5917.135907] Lustre: server umount lustre-OST0000 complete [ 5921.777964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.798669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5921.803262] Lustre: Skipped 1 previous similar message [ 5921.818263] Lustre: Skipped 2 previous similar messages [ 5926.884405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5926.893390] Lustre: Skipped 1 previous similar message [ 5927.249631] Lustre: server umount lustre-MDT0000 complete [ 5932.006073] LustreError: 170735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.029476] LustreError: 170735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5932.353756] LustreError: 170722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946130 with bad export cookie 16099395613910428157 [ 5932.356419] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5932.370355] LustreError: 170722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5933.127518] Lustre: server umount lustre-MDT0001 complete [ 5945.574222] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5948.853522] Key type lgssc unregistered [ 5949.096421] LNet: 174272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5949.106812] LNetError: 174272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5949.125398] LNet: Removed LNI 192.168.201.160@tcp [ 5949.750878] Key type .llcrypt unregistered [ 5949.752620] Key type ._llcrypt unregistered [ 5986.197223] Key type ._llcrypt registered [ 5986.198943] Key type .llcrypt registered [ 5986.271655] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5987.667402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5987.686480] alg: No test for adler32 (adler32-zlib) [ 5988.860689] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 5989.145237] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5990.848602] Key type lgssc registered [ 5992.542929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6002.497109] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6002.530146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.239386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.426255] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6014.433766] LustreError: 175981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.455181] LustreError: 175981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6018.984504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.552979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6025.259919] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6034.623746] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6042.882943] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6049.702647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6049.990324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.656942] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6068.948925] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6074.555611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6085.090555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6085.100154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6085.106551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.117680] Lustre: Skipped 1 previous similar message [ 6085.128211] Lustre: Skipped 1 previous similar message [ 6089.180468] Lustre: server umount lustre-OST0000 complete [ 6095.330153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.351585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.365507] Lustre: Skipped 1 previous similar message [ 6099.532354] Lustre: server umount lustre-MDT0000 complete [ 6100.451847] LustreError: 175976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.484972] LustreError: 175976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6103.356026] LustreError: 175961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946301 with bad export cookie 9498675121264830046 [ 6103.368664] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6104.168663] Lustre: server umount lustre-MDT0001 complete [ 6122.062042] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6126.620573] Key type lgssc unregistered [ 6127.025903] LNet: 179537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6127.034616] LNetError: 179537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6127.058839] LNet: Removed LNI 192.168.201.160@tcp [ 6128.250536] Key type .llcrypt unregistered [ 6128.258958] Key type ._llcrypt unregistered [ 6152.243264] Key type ._llcrypt registered [ 6152.249425] Key type .llcrypt registered [ 6152.389736] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6153.756795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6153.980867] alg: No test for adler32 (adler32-zlib) [ 6155.107121] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 6155.403121] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6157.088466] Key type lgssc registered [ 6158.148860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6168.890303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6168.920771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.507786] Lustre: *** cfs_fail_loc=135, val=0*** [ 6170.582873] LustreError: 181056:0:(obd_config.c:920:class_setup()) setup lustre-MDT0000 failed (-2) [ 6170.592457] LustreError: 181056:0:(obd_config.c:2201:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -2 [ 6170.610507] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6170.610507] [ 6170.633016] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6170.650882] LustreError: 181017:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 6170.671246] LustreError: 181017:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 6170.686211] LustreError: 181017:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 6170.699401] LustreError: 181017:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6170.803081] Lustre: server umount lustre-MDT0000 complete [ 6170.805298] LustreError: 181017:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6178.259635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.660635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.182885] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6189.052894] LustreError: 181428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.080835] LustreError: 181428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6192.217620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.652178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6197.081924] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6207.063622] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6213.876935] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6217.965287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.988466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6217.991875] Lustre: Skipped 1 previous similar message [ 6223.432280] LustreError: 181428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.803910] Lustre: server umount lustre-MDT0000 complete [ 6228.450756] LustreError: 181425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.478453] LustreError: 181425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6228.877787] LustreError: 181409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946426 with bad export cookie 3553504782521246210 [ 6228.880916] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.902827] LustreError: 181409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6229.792993] Lustre: server umount lustre-MDT0001 complete [ 6241.677555] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6244.487477] Key type lgssc unregistered [ 6245.034577] LNet: 183750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6245.038336] LNetError: 183750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6245.069485] LNet: Removed LNI 192.168.201.160@tcp [ 6246.222418] Key type .llcrypt unregistered [ 6246.223679] Key type ._llcrypt unregistered [ 6271.198029] Key type ._llcrypt registered [ 6271.202320] Key type .llcrypt registered [ 6271.342685] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6271.749631] Key type .llcrypt unregistered [ 6271.753501] Key type ._llcrypt unregistered [ 6289.232247] Key type ._llcrypt registered [ 6289.236807] Key type .llcrypt registered [ 6289.498524] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6291.266642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6291.394903] alg: No test for adler32 (adler32-zlib) [ 6292.376884] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 6292.710175] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6294.424483] Key type lgssc registered [ 6295.797612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6304.918786] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6304.939796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.736746] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6325.856413] LustreError: 185875:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 6352.608240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6359.570523] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6366.684736] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6373.341554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.947203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6374.072063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6378.978633] LustreError: 185900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.013673] LustreError: 185900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6379.773067] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6384.105471] LustreError: 185901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.211216] LustreError: 186913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.232732] LustreError: 186913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6389.077886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.474543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6393.188142] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6401.864415] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6408.335754] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6412.328723] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6425.058328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.067109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.084635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.589148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6425.597380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.613753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.691371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.701266] Lustre: Skipped 1 previous similar message [ 6435.816504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6435.824758] Lustre: Skipped 1 previous similar message [ 6439.392356] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6439.678445] Lustre: server umount lustre-OST0000 complete [ 6446.051447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.053620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6446.053652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6446.065407] Lustre: Skipped 1 previous similar message [ 6446.096168] Lustre: Skipped 1 previous similar message [ 6449.760041] Lustre: server umount lustre-MDT0000 complete [ 6451.173957] LustreError: 186909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6451.189992] LustreError: 186909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6453.733016] LustreError: 188537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946651 with bad export cookie 13177805398595168263 [ 6453.745326] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6453.757901] LustreError: 188537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6454.144816] Lustre: server umount lustre-MDT0001 complete [ 6465.702389] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6468.831942] Key type lgssc unregistered [ 6469.178648] LNet: 189888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.205461] LNetError: 189888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6469.241146] LNet: Removed LNI 192.168.201.160@tcp [ 6470.315795] Key type .llcrypt unregistered [ 6470.321546] Key type ._llcrypt unregistered [ 6493.645738] Key type ._llcrypt registered [ 6493.647334] Key type .llcrypt registered [ 6493.780907] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6495.286171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.325438] alg: No test for adler32 (adler32-zlib) [ 6496.360572] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 6496.593332] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6498.288292] Key type lgssc registered [ 6499.760846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6507.802356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6507.819443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6509.175530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6513.484791] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6518.759169] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.790063] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6522.806070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6523.477042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6528.247042] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6537.914915] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6545.072806] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6552.748961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6553.040103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.108905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6559.164300] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6566.928948] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6572.512395] Lustre: Failing over lustre-MDT0000 [ 6573.545060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.560769] LustreError: 192029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.568194] Lustre: Skipped 2 previous similar messages [ 6573.605318] LustreError: 192029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6574.721306] LustreError: 193930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.750488] Lustre: server umount lustre-MDT0000 complete [ 6578.658383] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6578.693037] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6583.779190] LustreError: 191300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.799071] LustreError: 191300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6589.920508] Lustre: 190609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946771/real 1788946771] req@000000009aa5822d x1875846568633216/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788946787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6589.937988] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.838810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.157930] LustreError: 191297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.177428] LustreError: 191297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6600.164688] LustreError: 190608:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d9cee1f8 x1875846568638848/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6600.303993] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.60@tcp (not set up) [ 6600.544767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6600.609449] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6604.807564] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6605.420130] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6605.802225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6605.829582] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6605.874539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6609.511236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6616.994338] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6623.212575] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6623.221189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.233634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.171744] Lustre: server umount lustre-OST0000 complete [ 6631.397938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.398593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.415366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.424854] Lustre: Skipped 1 previous similar message [ 6636.516720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6636.534017] Lustre: Skipped 1 previous similar message [ 6640.608947] Lustre: 190610:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946822/real 1788946822] req@000000001f1dad2a x1875846568662784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788946838 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6640.644359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6640.658226] Lustre: Skipped 1 previous similar message [ 6640.670896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6642.145296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6642.361661] Lustre: server umount lustre-MDT0000 complete [ 6642.720143] Lustre: 190609:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946824/real 1788946824] req@0000000053cec32f x1875846568663296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788946840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6645.751443] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.776160] LustreError: 191301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6646.401198] LustreError: 191280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946844 with bad export cookie 2201461625177546298 [ 6646.409251] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.413498] LustreError: 191280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6646.768992] Lustre: server umount lustre-MDT0001 complete [ 6659.203704] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6663.372606] Key type lgssc unregistered [ 6663.755540] LNet: 196185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6663.762429] LNetError: 196185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6663.790276] LNet: Removed LNI 192.168.201.160@tcp [ 6665.176789] Key type .llcrypt unregistered [ 6665.180099] Key type ._llcrypt unregistered [ 6693.582876] Key type ._llcrypt registered [ 6693.592167] Key type .llcrypt registered [ 6693.737615] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6695.617751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6695.650774] alg: No test for adler32 (adler32-zlib) [ 6696.803596] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 6697.062786] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6698.728192] Key type lgssc registered [ 6700.371681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6709.988185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6710.032272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.596147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6717.206543] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6722.030254] LustreError: 197593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6722.040538] LustreError: 197593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6726.993125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.138269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6727.290568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6731.911649] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6739.848534] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6745.660587] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6752.045667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6752.403093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.532811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6758.189390] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6767.101433] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6770.535336] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6775.552895] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6794.233769] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6808.035595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6808.042990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6808.055508] Lustre: Skipped 1 previous similar message [ 6812.245318] Lustre: server umount lustre-OST0000 complete [ 6818.280581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.300358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6818.302549] Lustre: Skipped 1 previous similar message [ 6822.601994] Lustre: server umount lustre-MDT0000 complete [ 6823.393729] LustreError: 198792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.414643] LustreError: 198792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6826.806847] LustreError: 197574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947024 with bad export cookie 3603969857943383286 [ 6826.818478] LustreError: 197574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6826.818873] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.106095] Lustre: server umount lustre-MDT0001 complete [ 6838.837988] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6841.702319] Key type lgssc unregistered [ 6841.962409] LNet: 201462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6841.966883] LNetError: 201462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6842.983668] LNet: Removed LNI 192.168.201.160@tcp [ 6843.721347] Key type .llcrypt unregistered [ 6843.722944] Key type ._llcrypt unregistered [ 6867.052089] Key type ._llcrypt registered [ 6867.054259] Key type .llcrypt registered [ 6867.157101] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6868.458393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6868.480541] alg: No test for adler32 (adler32-zlib) [ 6869.518652] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 6869.732168] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6871.352221] Key type lgssc registered [ 6872.551640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6880.929252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6880.941921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.664648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.885468] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6893.031256] LustreError: 202724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6893.043130] LustreError: 202724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6897.384390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.662192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.036369] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6912.085755] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6918.174662] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.286073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6926.877729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.951592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6933.919158] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6942.247852] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6949.478947] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6962.359908] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6973.586774] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6979.903825] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6988.772133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.773805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.779435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6988.808888] Lustre: Skipped 1 previous similar message [ 6993.889784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6993.892320] Lustre: Skipped 1 previous similar message [ 6994.751824] Lustre: server umount lustre-OST0000 complete [ 6999.017031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.036519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6999.036633] Lustre: Skipped 1 previous similar message [ 6999.060276] Lustre: Skipped 2 previous similar messages [ 7004.138123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7004.144202] Lustre: Skipped 1 previous similar message [ 7009.250133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.346178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7013.535494] Lustre: server umount lustre-MDT0000 complete [ 7014.388391] LustreError: 203451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.406749] LustreError: 203451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7018.040728] LustreError: 202704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947215 with bad export cookie 11335644334300701709 [ 7018.054961] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7018.077824] LustreError: 202704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7018.586100] Lustre: server umount lustre-MDT0001 complete [ 7030.929645] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7034.050290] Key type lgssc unregistered [ 7034.404134] LNet: 207602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7034.411326] LNetError: 207602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7034.428370] LNet: Removed LNI 192.168.201.160@tcp [ 7035.338142] Key type .llcrypt unregistered [ 7035.339596] Key type ._llcrypt unregistered [ 7062.836787] Key type ._llcrypt registered [ 7062.839979] Key type .llcrypt registered [ 7063.004679] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7064.579889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7064.636869] alg: No test for adler32 (adler32-zlib) [ 7065.878696] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 7066.199627] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7067.936309] Key type lgssc registered [ 7069.371749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7078.730125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7078.750256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.350653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7085.323477] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7090.662361] LustreError: 209012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.692617] LustreError: 209012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7094.293364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.640797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7100.077497] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7108.108778] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7114.985473] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7121.956635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7122.315235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.365782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 7128.803627] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7137.305978] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7150.056479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7150.063623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.072687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7150.075205] Lustre: Skipped 1 previous similar message [ 7155.171630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7155.177434] Lustre: Skipped 2 previous similar messages [ 7156.302140] Lustre: server umount lustre-OST0000 complete [ 7160.295705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.304235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.304624] Lustre: Skipped 1 previous similar message [ 7160.326231] Lustre: Skipped 1 previous similar message [ 7165.414971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7165.425452] Lustre: Skipped 1 previous similar message [ 7170.530314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7170.547406] Lustre: Skipped 1 previous similar message [ 7174.112145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7174.374957] Lustre: server umount lustre-MDT0000 complete [ 7175.650987] LustreError: 209012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7175.667454] LustreError: 209012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7178.356369] LustreError: 211946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947376 with bad export cookie 7906717742381862765 [ 7178.367671] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7178.374032] LustreError: 211946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7178.835870] Lustre: server umount lustre-MDT0001 complete [ 7190.369340] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7193.039898] Key type lgssc unregistered [ 7193.352133] LNet: 212699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7193.374712] LNetError: 212699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7193.394844] LNet: Removed LNI 192.168.201.160@tcp [ 7194.392729] Key type .llcrypt unregistered [ 7194.395914] Key type ._llcrypt unregistered [ 7212.432429] Key type ._llcrypt registered [ 7212.434309] Key type .llcrypt registered [ 7212.526508] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7213.181439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7213.194672] alg: No test for adler32 (adler32-zlib) [ 7214.197231] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 7214.563830] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7216.264181] Key type lgssc registered [ 7217.441748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7225.040763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7225.055328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.555426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7231.219438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7237.101920] LustreError: 213960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.126058] LustreError: 213960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7239.672387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.112904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7244.998474] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7253.210230] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7260.134144] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7267.592979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.023539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7272.133701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7274.301996] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7282.389926] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7298.017409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7298.023374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7298.042856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7300.069597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7300.074225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7300.075084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.120688] Lustre: Skipped 1 previous similar message [ 7302.977205] Lustre: server umount lustre-OST0000 complete [ 7306.723300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7306.738252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.773923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.313261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.315239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7313.093477] Lustre: server umount lustre-MDT0000 complete [ 7315.426396] LustreError: 213959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.456364] LustreError: 213959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7317.460380] LustreError: 213940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947515 with bad export cookie 15167789516174537721 [ 7317.462682] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7317.483088] LustreError: 213940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7317.970156] Lustre: server umount lustre-MDT0001 complete [ 7332.612148] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7335.526698] Key type lgssc unregistered [ 7335.765563] LNet: 217641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7335.774213] LNetError: 217641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7335.787589] LNet: Removed LNI 192.168.201.160@tcp [ 7336.448798] Key type .llcrypt unregistered [ 7336.451530] Key type ._llcrypt unregistered [ 7354.945827] Key type ._llcrypt registered [ 7354.949502] Key type .llcrypt registered [ 7355.111530] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7356.226111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7356.334223] alg: No test for adler32 (adler32-zlib) [ 7357.641817] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 7357.971610] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7359.744161] Key type lgssc registered [ 7361.033909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.890413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7368.911224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7370.305933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.245730] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7380.462417] LustreError: 218905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.475256] LustreError: 218905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7384.665572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.956608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.622421] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7398.034840] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7405.080878] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7413.045548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.436820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.482616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7419.714094] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7426.737575] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7435.192925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.254578] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7435.448947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7441.624236] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7449.187469] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7450.649458] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7462.232275] Lustre: Permanently deactivating lustre-OST0001 [ 7462.241386] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7472.853563] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7483.932098] Lustre: Permanently reactivating lustre-OST0001 [ 7483.953703] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7483.956824] Lustre: Skipped 2 previous similar messages [ 7493.690327] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.702049] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7498.729449] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7499.211970] Lustre: server umount lustre-OST0001 complete [ 7503.844297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.856851] LustreError: 220728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.860482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7503.872383] Lustre: Skipped 1 previous similar message [ 7508.971850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7508.972506] LustreError: 221404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.982488] Lustre: Skipped 2 previous similar messages [ 7514.080525] LustreError: 222840:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7514.084128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.085708] LustreError: 221404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7514.092350] Lustre: Skipped 1 previous similar message [ 7514.123284] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.162027] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7517.511094] Lustre: server umount lustre-OST0000 complete [ 7534.562919] LustreError: 223045:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0000: unknown param osc.active=1 [ 7538.658722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7538.670077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.694829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7540.943479] Lustre: server umount lustre-MDT0000 complete [ 7544.804724] LustreError: 219634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.818651] LustreError: 219634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7544.863023] LustreError: 220752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788947742 with bad export cookie 15661818746945983824 [ 7544.866242] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7544.873700] LustreError: 220752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7545.309779] Lustre: server umount lustre-MDT0001 complete [ 7556.883863] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7559.797473] Key type lgssc unregistered [ 7560.051831] LNet: 223999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7560.068764] LNetError: 223999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7561.130726] LNet: Removed LNI 192.168.201.160@tcp [ 7562.282190] Key type .llcrypt unregistered [ 7562.285326] Key type ._llcrypt unregistered [ 7602.910616] Key type ._llcrypt registered [ 7602.913930] Key type .llcrypt registered [ 7603.026287] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7604.025091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7604.080310] alg: No test for adler32 (adler32-zlib) [ 7605.343774] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 7605.575625] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7607.252387] Key type lgssc registered [ 7608.547207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7616.897434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7616.926641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7618.242672] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7618.291539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7618.649858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7623.793326] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7633.044857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.333343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7633.409290] Lustre: 226200:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7633.715664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7638.767528] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7646.472150] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7651.911940] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7658.485187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7658.681203] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7658.879154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7665.821743] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7668.888930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7673.801711] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7676.527514] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7682.917314] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7692.513208] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7700.265283] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7709.256076] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7727.076459] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7727.080987] Lustre: Skipped 1 previous similar message [ 7763.258665] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7763.266707] Lustre: Skipped 3 previous similar messages [ 7831.660746] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7831.664324] Lustre: Skipped 7 previous similar messages [ 7868.387332] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.394902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.406766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7868.926395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.938076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.943220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7872.598552] Lustre: server umount lustre-OST0000 complete [ 7879.137298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.159455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7879.168844] Lustre: Skipped 1 previous similar message [ 7883.181710] Lustre: server umount lustre-MDT0000 complete [ 7886.926336] LustreError: 228768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948084 with bad export cookie 7467068838192744425 [ 7886.927348] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.931946] LustreError: 228768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7887.328312] Lustre: server umount lustre-MDT0001 complete [ 7898.564954] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7902.093973] Key type lgssc unregistered [ 7902.512857] LNet: 231071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7902.521740] LNetError: 231071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7902.539938] LNet: Removed LNI 192.168.201.160@tcp [ 7903.880409] Key type .llcrypt unregistered [ 7903.882125] Key type ._llcrypt unregistered [ 7931.481236] Key type ._llcrypt registered [ 7931.484713] Key type .llcrypt registered [ 7931.669316] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7933.115797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7933.349970] alg: No test for adler32 (adler32-zlib) [ 7934.475808] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 7934.846172] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7936.640192] Key type lgssc registered [ 7938.228648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7946.191102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7946.207736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.660890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.710952] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7957.989832] LustreError: 232484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.998252] LustreError: 232484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7962.173872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.669219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7967.078799] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7975.568066] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7982.637670] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7990.050900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.643486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7992.705300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7998.101443] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 8007.211435] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8023.523652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8023.534064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8023.550137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.546573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.557188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8024.559868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.430940] Lustre: server umount lustre-OST0000 complete [ 8034.279346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8034.291842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.306974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8034.313673] Lustre: Skipped 1 previous similar message [ 8039.909268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.922079] Lustre: Skipped 2 previous similar messages [ 8045.039321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.050352] Lustre: Skipped 1 previous similar message [ 8048.096293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8048.420414] Lustre: server umount lustre-MDT0000 complete [ 8050.157464] LustreError: 232483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.187556] LustreError: 232483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8052.159153] LustreError: 233217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788948249 with bad export cookie 2892381432607186080 [ 8052.166853] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.176419] LustreError: 233217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8052.663431] Lustre: server umount lustre-MDT0001 complete [ 8063.560564] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 8066.564549] Key type lgssc unregistered [ 8066.905386] LNet: 236174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8066.911820] LNetError: 236174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8067.943978] LNet: Removed LNI 192.168.201.160@tcp [ 8069.172583] Key type .llcrypt unregistered [ 8069.174882] Key type ._llcrypt unregistered [ 8095.154844] Key type ._llcrypt registered [ 8095.164282] Key type .llcrypt registered [ 8095.316487] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8097.014584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8097.048681] alg: No test for adler32 (adler32-zlib) [ 8098.321779] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 8098.697244] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 8100.432203] Key type lgssc registered [ 8101.907815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8373.330520] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 8375.933364] Key type lgssc unregistered [ 8376.210573] LNet: 238197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8376.218470] LNetError: 238197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8376.248785] LNet: Removed LNI 192.168.201.160@tcp [ 8377.100603] Key type .llcrypt unregistered [ 8377.102530] Key type ._llcrypt unregistered [ 8404.799572] Key type ._llcrypt registered [ 8404.802230] Key type .llcrypt registered [ 8404.937225] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 8425.314821] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8426.752926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8426.781683] alg: No test for adler32 (adler32-zlib) [ 8427.764840] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 8428.125435] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 8429.776361] Key type lgssc registered [ 8430.812471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8465.304928] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8111 sec ========== 06:11:00 (1788948660) [ 8466.796435] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:11:02 (1788948662) === [ 8469.694652] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:11:05 (1788948665) === [ 8507.181227] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 8509.697225] Key type lgssc unregistered [ 8509.937790] LNet: 243436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8509.943436] LNetError: 243436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8509.963166] LNet: Removed LNI 192.168.201.160@tcp [ 8510.679518] Key type .llcrypt unregistered [ 8510.683171] Key type ._llcrypt unregistered